protected override void Seed(Fitlife.Domain.Concrete.EFDBContext context)
{
List<List<string>> foodweights = GetLines(basePath + "FoodWeights.txt");
int counter = 0;
foodweights.ForEach(line =>
{
FoodWeights newVal = new FoodWeights()
{
FoodCode = int.Parse(line[0]),
PortionCode = int.Parse(line[1]),
PortionWeight = decimal.Parse(line[2])
};
context.FoodWeights.Add(newVal);
if (++counter == 1000)
{
counter = 0;
context.SaveChanges();
}
});
}
Above method is used to populate my database. But it takes 50 seconds for 1000 entries i have a file with 470k entries, how can i improve performance i am using entity framework and this method is called when i do
PM> update-database
with Package manager. i need similar functionality, i am very new to asp.net and entity framework any guidance will be appreciated thanks.
PS: Is it ok to take 50 seconds for 1000 entries or am i doing something wrong.
The Seed method runs every time the application starts, so the way you have coded it will attempt to add the FoodWeights over and over again. EF have provided the AddOrUpdate as a convenient method to prevent that but it is really not appropriate for bulk inserts.
You could use sql directly on the database - and if you are using sql server that sql could be 'BULK INSERT'.
I would put the sql in an Up migration because you probably only want to run the insert once from a known state, and it avoids having to worry about the efficiency of the context and tracking changes etc.
There is example code and more information here: how to seed data using sql files
Related
Because of the performance benefits, I am inserting data by executing raw sql command.
All data are properly inserted into database, but the repository is not aware of them being inserted. Basically, usersBeforeInsert and usersAfterInsert contain the same 3 records.
Also, after application restart, repository is still not aware of users inserted with ExecuteSqlRaw(), retrieving only users that existed prior to ExecuteSqlRaw().
Does anybody knows how to make _userRepository retrieve all data from db? Btw, I am using asp.net boilerplate project.
Here is the code sample:
var usersBeforeInsert = await _userRepository.GetAllListAsync(); // 3
var commandText = GenerateInsertUsersSqlScript(users);
var context = _userRepository.GetDbContext();
var rowsAffected = context.Database.ExecuteSqlRaw(commandText); // 85000
context.SaveChange()
var usersAfterInsert = await _userRepository.GetAllListAsync(); // 3
Your culprit is this line
var usersBeforeInsert = await _userRepository.GetAllListAsync();
Because the framework assume nothing has changed between this line and this line
var usersAfterInsert = await _userRepository.GetAllListAsync();
You fix this by not calling the list until after the raw insert, or you can clear the changeset.
the easiest way to clear the changeset is to new up a new _userRepository, but I guess it most likely is injected.
In a .NET Core project I have the following model:
public class WrapperContext : DbContext
{
public DbSet<WrappedFunction> Functions { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder builder)
{
builder.UseSqlite("Data Source=functions.db");
}
}
public class WrappedFunction
{
[Key]
public int FunctionId { get; set; }
public String FunctionName { get; set; }
public String AssemblyPath { get; set; }
public String FactoryName { get; set; }
}
As you can see it is an Sqlite database.
I have added the initial migration and updated the database. Before running my test application I have populated the Functions table with a single record. I can open the DB with SqliteManager to see the record is there.
I then try to access the DB with the following function:
static MUinfo getInfoFor(String command)
{
UFInfo rv = null;
using (var ctx = new WrapperContext()) // <---- record disappears here
{
foreach (var fn in ctx.Functions)
{
if (fn.FunctionName.Equals(command))
{
rv = new UFInfo()
{
AssemblyPath = fn.AssemblyPath,
FactoryName = fn.FactoryName,
CommandName = command
};
}
}
}
if (rv != null) return rv;
return "No Info for " + command;
}
However, when stepping through the debugger I see that the foreach loop never executes as there are no records in the DB. When I stop before executing the foreach I can verify (using SqliteManager) that my single record has been deleted from the database.
So, it would appear that the DbContext object is somehow deleting the data from the database. Why would that be and what have I done wrong here? I am fairly new to EntityFramework so I may have done something obvious, but it isn't obvious to me.
One additional bit of info here. The project that does the db access is a library project. I have to copy the DB over to the build directory of the application before populating it and running the application. I don't know if that makes a difference or not.
Edit 6/19/17 (18:51):
OK. A bit more information (thanks to #StevePy). Nunit3 seems to conflict with EFCore (though I don't get any errors on restore) so I couldn't create a unit test for this. So I inserted a using (var ctx...) above the one in the listing above and inserted a couple of dummy records. Then I exited that using block and when I entered the one to traverse the records they were there.
However, I then commented out the dummy insertions and reran my test. And, once again, both records disappeared. So there is something very weird going on here with EFCore.
Edit 6/20/17 (15:30):
Well, I'm still not sure what is going on but there is an odd interaction between EFCore and Microsoft.Data.Sqlite.
I decided to remove all references to EFCore and simply use SQL queries on the DB. I did this without recreating the DB (so I was using the one already created by EFCore). I noticed the exact same behavior when I created a new SqliteConnection without using EFCore.
In desperation I deleted the DB and recreated it from scratch using Microsoft.Data.Sqlite. This DB didn't have any of the EFCore information in it, obviously. I then populated the DB with some test records and went to use it. No more disappearing records.
Apparently there was something very strange in the way EFCore set up the database in the first place that caused the issue. But I don't have any idea what it was.
What is likely happening is that you are using CodeFirst /w EF, but taking a DB first approach when it comes to testing your new code. EF tracks schema changes within the database and my guess is that by you creating the table ahead of time it does not know that the schema is vetted, so it drops and recreates it on context start-up.
The fix here should be to create a "stub" test that populates a test record one time using your EF model. From there you should have a table that EF recognizes against that context and accepts. From there you can create test records in whatever editor for testing purposes.
SQLite has several limitations when it comes to schema migration that you probably will want to consider as well. (see: https://learn.microsoft.com/en-us/ef/core/providers/sqlite/limitations)
You might be better off setting up DB first, telling EF how to map to an existing SQLite schema rather than trying to use Code-First as migration is pretty limited for that DB engine.
I developed a client .NET WinForms application. This application is used to populate a database with thousands of records. For the data layer I used EF6.
When I work and run the application locally every thing works as expected. Insertions are very fast (over 500000 records in about 2 mins).
Now I'm trying to use a remote database on a hosted server and I notice that insertions are very very slow (less than 500 records in about 2 mins). This means 1000 times slower than locally.
If I try to insert 500 records in the remote database using SQL Server Management Studio the operation is completed in less than 1 second.
Is the problem on my client application?
Here the insert function:
public void SetDemoDimCustomer()
{
DWContext dc = null;
try
{
dc = DWContext.Create(SqlServerInstance, DbName);
dc.Configuration.AutoDetectChangesEnabled = false;
dc.Database.ExecuteSqlCommand("DELETE FROM DimCustomer");
dc.Database.ExecuteSqlCommand("DBCC CHECKIDENT ('DimCustomer', RESEED, 0)");
DimCustomer objCustomer;
List<DimCustomer> lstDemoCustomers = new List<DimCustomer>();
int length = 100;
for (int i = 0; i < length; i++)
{
objCustomer = new DimCustomer();
objCustomer.Name = "Customer " + (i + 1);
objCustomer.CustomerBKey = (i + 1).ToString();
lstDemoCustomers.Add(objCustomer);
}
dc.DimCustomer.AddRange(lstDemoCustomers);
dc.SaveChanges();
}
catch (Exception)
{
throw;
}
finally
{
if (dc != null)
{
dc.Dispose();
}
}
}
I tried to use Linq-to-SQL instead of EF6 but the result is the same. Maybe is not a specific EF6 problem.
Some infos about the remote system:
OS: Windows Server 2012
RDBMS: SQL Server 2014 Express
Thanks in advance.
UPDATE AFTER SOME TESTS WITH BULKINSERT
Ok here the results of my first tests with BulkInsert:
100 records -> EF6 AddRange: 9 sec. / EF6 BulkInsert: 1 sec.
1000 records -> EF6 AddRange: 1:27 min. / EF6 BulkInsert: 1 sec. (wow!)
10000 records -> EF6 AddRange: 14:39 min. / EF6 BulkInsert: 4 sec. (wooooow!)
Now, of course, the EF6 BulkInsert package is part of my project.
Looks like most time is spend on the network waiting for a round-trip to complete. EF cannot be made to batch inserts (yet). So you cannot use EF for inserts here.
Investigate the typical solutions to this problem (TVPs and SqlBulkCopy).
The dispose pattern you are using is not a good choice. Just wrap dc in ´using` and delete all exception handling.
As suggested, SqlBulkCopy is your best bet, however there is an Interesting Nuget Package which does BulkInsert for Entity Framework:
https://efbulkinsert.codeplex.com/
I am using ADO.NET and trying to update a column value in a table in a SQL Server 2008 database. However the value is not updating. Even though it's update when I debug it in C# code. Here is my code:
using (ADONETClass context = new ADONETClass())
{
List<Invoices> list = context.Invoices.ToList<Invoices>();
foreach (Invoices b in list)
{
b.status = 0;
}
}
Now I debugged the code and saw that context.Invoices.ToList<Invoices>()[0].status is indeed set to zero. But when the program finished running, I open the SQL Server Management Studio and status value there was still 0. Not sure what is going on? Am I missing something?
Thanks
Varun
Are you calling context.SaveChanges()?
Are you using Entity Framework?
I think you're missing a call to context.SaveChanges
In EntityFramework the changes are all just in memory and not written to database until you call SaveChanges
I'm using SMO to execute a batch SQL script. In Management Studio, the script executes in about 2 seconds. With the following code, it takes about 15 seconds.
var connectionString = GetConnectionString();
// need to use master because the DB in the connection string no longer exists
// because we dropped it already
var builder = new SqlConnectionStringBuilder(connectionString)
{
InitialCatalog = "master"
};
using (var sqlConnection = new SqlConnection(builder.ToString()))
{
var serverConnection = new ServerConnection(sqlConnection);
var server = new Server(serverConnection);
// hangs here for about 12 -15 seconds
server.ConnectionContext.ExecuteNonQuery(sql);
}
The script creates a new database and inserts a few thousand rows across a few tables. The resulting DB size is about 5MB.
Anyone have any experience with this or have a suggestion on why this might be running so slowly with SMO?
SMO does lots of weird .. stuff in the background, which is a price you pay for ability to treat server/database objects in an object-oriented way.
Since you're not using the OO capabilites of SMO, why don't you just ignore SMO completely and simply run the script through normal ADO?
The best and fastest way to upload records into a database is through SqlBulkCopy.
Particularly when your scripts are ~1000 records plus - this will make a significant speed improvement.
You will need to do a little work to get your data into a DataSet, but this can easily be done using the DataSet xml functions.