When I try to query a PostgreSQL database through EntityFramework6.Npgsql with the following code:
using (MyDbContext context = new MyDbContext())
{
var res = (from b in context.mytable select new { b.Name, b.Age });
foreach (var row in res)
{
Console.WriteLine(row.Name + " - " + row.Age);
}
}
I get a timeout exception after fetching few lines with the following error:
[Npgsql.NpgsqlException] : {"57014: canceling statement due to
statement timeout"}
Message: 57014: canceling statement due to
statement timeout
When I execute the same operation while fetching all the data to a List, the code works fine:
using (MyDbContext context = new MyDbContext())
{
var res = (from b in context.mytable select new { b.Name, b.Age }).ToList();
foreach (var row in res)
{
Console.WriteLine(row.Name + " - " + row.Age);
}
}
I suspect that it is related to the way PostgreSQL manages its connection pool but I don't know how I could handle it correctly through Entity Framework.
This is probably related to the way Npgsql manages timeouts. In current versions, Npgsql sets the PostgreSQL statement_timeout variable which causes PostgreSQL to generate a timeout error after some time. The problem with this method is that statement_timeout is unreliable for this: it includes network time, client processing time, etc. so too much time spent on the client could make the server generate the error.
In your example, calling ToList() means that you immediately download all results, rather than iterate over them little by little. I do admit it's strange that such short client processing (i.e. Console.WriteLine) could introduce a delay sufficient to trigger a backend timeout (what is the command timeout set to?).
Note that the next major version of Npgsql will remove backend timeouts entirely because of the unreliable nature of statement_timeout - see https://github.com/npgsql/npgsql/issues/689. For now you can manually disable backend timeouts by setting the Backend Timeouts connection string parameter to false (see http://www.npgsql.org/doc/3.0/connection-string-parameters.html).
Related
I develop a console application to transfert data in a API. The application is called via a SQL Trigger.
I use a dbcontext to get the data. it worked, but suddenly it doesn't work anymore.
here my code :
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer(
"Data Source=.\SQLEXPRESS;Initial Catalog=mydatabase;User ID=sa;Password=password;;Trust Server Certificate=true;Encrypt=true;Connection Timeout=300",
providerOptions =>
{
providerOptions.CommandTimeout(180);
});
}
then in my program.cs
using (DB_DONNNESContext context = new DB_DONNNESContext())
{
ListeLot lot = new ListeLot();
try
{
lot = context.ListeLots.Where(e => e.Id == Id).First();
Log.Information("Get lot " + Lot.Id);
Log.CloseAndFlush();
return;
}
catch (Exception ex)
{
Log.Error($"Error cannot get lot {Id} " + ex.Message + " " + " / "+ context.ContextId);
Log.CloseAndFlush();
return;
}
}
what's wrong ? my query is very simple... yesterday it's worked once and then I had this message. I have 6Go on the hard disk.
When I test in debug mode, it work. but when I Try to update via a sql query I have this message in the log file
Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. / Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade
It's strange cause I get a contextID. (I try in Loggin it)
I really need your help :)
have a nice day.
You may be encountering a deadlock. Querying a table as part of a Trigger could be potentially dangerous depending on your database isolation level.
If your code represents all you need to do, then it would be completely unnecessary and likely contributing to a deadlock scenario.
lot = context.ListeLots.Where(e => e.Id == Id).First();
Log.Information("Get lot " + Lot.Id);
This code is telling EF to load an entity from the database, and there appears to be a typo or a reference issue given the "lot" being loaded is "lot" but your Log.Information call is referencing "Lot". This could be a typo getting your code into the StackOverflow question, or your code is attempting to reference a module level/global property called "Lot". C# is case sensitive but you can run into issues like this quite easily when reusing variable names varying by case.
If you just want to check if a Lot record exists without loading the entire thing into memory (triggering a read lock for the row which may be deadlocking with the trigger or other DB operations) then instead use the following:
var lotExists = context.ListeLots.Any(e => e.Id == Id);
if(lotExists)
Log.Information($"Get lot {Id}");
else
Log.Warning($"Lot {Id} not found.");
Log.CloseAndFlush();
Rather than loading the Log entity, doing an Any check will just return if the value exists. This should amount to an index scan/seek and unlikely to deadlock.
Edit: If you need to get a couple of columns from a query and want to improve the chance to avoid the deadlock, one option would be to add those columns as included fields on an index. For instance if you want to get a Name column from a lot, create an index on ID with an included column for Name. Then when fetching your Id and Name from the ListeLots:
var lot = context.ListeLots
.Where(e => e.Id == Id)
.Select(e => new { e.Id, e.Name })
.SingleOrDefault();
When EF composes this to SQL and executes it on the server, SQL Server should optimize this to pull the values from the index rather than the table. A caveat though if you are searching for a data row that has been updated (I.e. you are searching for a row that the trigger is executing after an update on) then this will most likely still deadlock, especially if the Name had been updated. Indexes can help bypass deadlock scenarios due to row locks, but they aren't immune to them. This can generally improve query performance, but comes at a cost of storage/memory use on the DB server as well as update performance costs. Generally it can be a good trade off for a small number of commonly queried, small fields. However, as soon as you request a column not in the index it will return to the table data.
When creating a report I have to execute 3 queries that involve separated entities of the same context. Because they are quite heavy ones I decided to use the .ToListAsync(); in order to have them run in parallel, but, to my surprise, I get a exception out of it...
What is the correct way to perform queries in parallel using EF 6? Should I manually start new Tasks?
Edit 1
The code is basically
using(var MyCtx = new MyCtx())
{
var r1 = MyCtx.E1.Where(bla bla bla).ToListAsync();
var r2 = MyCtx.E2.Where(ble ble ble).ToListAsync();
var r3 = MyCtx.E3.Where(ble ble ble).ToListAsync();
Task.WhenAll(r1,r2,r3);
DoSomething(r1.Result, r2.Result, r3.Result);
}
The problem is this:
EF doesn't support processing multiple requests through the same DbContext object. If your second asynchronous request on the same DbContext instance starts before the first request finishes (and that's the whole point), you'll get an error message that your request is processing against an open DataReader.
Source: https://visualstudiomagazine.com/articles/2014/04/01/async-processing.aspx
You will need to modify your code to something like this:
async Task<List<E1Entity>> GetE1Data()
{
using(var MyCtx = new MyCtx())
{
return await MyCtx.E1.Where(bla bla bla).ToListAsync();
}
}
async Task<List<E2Entity>> GetE2Data()
{
using(var MyCtx = new MyCtx())
{
return await MyCtx.E2.Where(bla bla bla).ToListAsync();
}
}
async Task DoSomething()
{
var t1 = GetE1Data();
var t2 = GetE2Data();
await Task.WhenAll(t1,t2);
DoSomething(t1.Result, t2.Result);
}
As a matter of interest, when using EF Core with Oracle, multiple parallel operations like the post here using a single DB context work without issue (despite Microsoft's documentation). The limitation is in the Microsoft.EntityFrameworkCore.SqlServer.dll driver, and is not a generalized EF issue. The corresponding Oracle.EntityFrameworkCore.dll driver doesn't have this limitation.
Check out https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/enabling-multiple-active-result-sets
From the documentation:
Statement interleaving of SELECT and BULK INSERT statements is
allowed. However, data manipulation language (DML) and data definition
language (DDL) statements execute atomically.
Then your above code works and you get the performance benefits for reading data.
I know this question has been asked many times however none of the answers fit my issue.
I have a thread timer firing every 30 seconds that queries a MSSQL db that is under heavy load. If i need to update the data in the console app that i'm using i use Linq To Sql to update the data stored in memory.
My problem is sometimes I get the error ExecuteReader requires an open and available Connection.
The code from the thread timer fires a Thread.Run(reload());
The connection string is
//example code
void reload(...
string connstring = string.Format("Data Source={0},{1};Initial Catalog={2};User ID={3};Password={4};Application Name={5};Connect Timeout=120;MultipleActiveResultSets=True;Max Pool Size=1524;Pooling=true;"
settings = new ConnectionStringSettings("sqlServer", connstring, "System.Data.SqlClient");
using (var tx = new TransactionScope(TransactionScopeOption.Required,
new TransactionOptions() { IsolationLevel = IsolationLevel.ReadUncommitted }))
{
using (SwitchDataDataContext data = new SwitchDataDataContext(settings.ConnectionString))
{
data.CommandTimeout = 560;
then i do many linqtosql searches. The exceptions happen from time to time but not always on the same query's. it's like the connections is opened and is forced closed.
Sometimes the exceptions says the current status is Open, Closed, Connecting. I add a larger ThreadPool to the SQL db but nothing seems to help.
i also have ADO in other parts of the program without any issues.
I believe that your problem is that the transaction scope also has a timeout. The default timeout is 1 minute according to this answer. So the transaction times out long before your command does (560 seconds = 9.3 minutes or so) . You will need to set the timeout property in the instance of the TransactionOptions object you are creating
new TransactionOptions()
{
IsolationLevel = IsolationLevel.ReadUncommitted,
Timeout = new TimeSpan(0,10,0) /* 10 Minutes */
}
You can verify that is indeed the issue by setting the TransactionScope timeout to a small value to force it to timeout.
I changed Linq To Sql to Entity Framework and received the same type of message. I believe the issues is lazy Loading. I was using the collections before it was ready on a different thread. I just added .Include("Lab") to my collection to load the entire collection and it seems to of fixed the issue.
There are many articles here on EF taking a long time to save, but I've looked through them and used their answers and still seem to get very slow results.
My code looks like so:
using (MarketingEntities1 db = new MarketingEntities1())
{
//using (var trans = db.Database.BeginTransaction(IsolationLevel.ReadUncommitted))
//{
int count = 0;
db.Configuration.AutoDetectChangesEnabled = false;
db.Configuration.ValidateOnSaveEnabled = false;
while (count < ranges.Count)
{
if (bgw != null)
{
bgw.ReportProgress(0, "Saving count: " + count.ToString());
}
db.Set<xGeoIPRanx>().AddRange(ranges.Skip(count).Take(BATCHCOUNT));
db.SaveChanges();
count+=BATCHCOUNT;
}
//trans.Commit();
//}
}
Each batch takes 30+ seconds to complete. BatchCount is 1000. i know EF isn't that slow. You can see that I've stopped using transaction, I've taken tracking off, none of it seemed to help.
Some more info:
xGeoIpRanx is an empty table, with no PK(I'm not sure how much it would help). I'm trying to insert about 10 mil ranges.
Edit:
i feel stupid but im trying to use bulkInsert and i keep getting this entity doesnt exist errors, i look at this code
using (var ctx = GetContext())
{
using (var transactionScope = new TransactionScope())
{
// some stuff in dbcontext
ctx.BulkInsert(entities);
ctx.SaveChanges();
transactionScope.Complete();
}
}
What is "entities" I tried a list of my entities, that doesnt work, what data type is that?
nvm it works as expected it was a strange error due to how i generated the edmx file
Pause the debugger 10 times under load and look at the stack including
external code. Where does it stop most often?
.
Its taking a long time on the .SaveChanges(). just from some quick tests, ADO.net code
That means network latency and server execution time are causing this. For inserts server execution time is usually not that high. You cannot do anything about network latency with EF because it sends one batch per insert. (Yes, this is a deficiency of the framework.).
Don't use EF for bulk work. Consider using table-values parameters or SqlBulkCopy or any other means of bulk inserting such as Aducci's proposal from the comments.
I need some advice regarding an application I wrote. The issues I am having are due to my DAL and connections to my SQL Server 2008 database not being closed, however I have looked at my code and each connection is always being closed.
The application is a multithreaded application that retrieves a set of records and while it processes a record it updates information about it.
Here is the flow:
The administrator has the ability to set the number of threads to run and how many records per thread to pull.
Here is the code that runs after they click start:
Adapters are abstractions to my DAL here is a sample of what they look like:
public class UserDetailsAdapter: IDataAdapter<UserDetails>
{
private IUserDetailFactory _factory;
public UserDetailsAdapter()
{
_factory = new CampaignFactory();
}
public UserDetails FindById(int id){
return _factory.FindById(id);
}
}
As soon as the _factory is called it processes the SQL and immediately closes the connection.
Code For Threaded App:
private int _recordsPerthread;
private int _threadCount;
public void RunDetails()
{
//create an adapter instance that is an abstration
//of the data factory layer
var adapter = new UserDetailsAdapter();
for (var i = 1; i <= _threadCount; i++)
{
//This adater makes a call tot he databse to pull X amount of records and
//set a lock filed so the next set of records that are pulled are differnt.
var details = adapter.FindTopDetailsInQueue(_recordsPerthread);
if (details != null)
{
var parameters = new ArrayList {i, details};
ThreadPool.QueueUserWorkItem(ThreadWorker, parameters);
}
else
{
break;
}
}
}
private void ThreadWorker(object parametersList)
{
var parms = (ArrayList) parametersList;
var threadCount = (int) parms[0];
var details = (List<UserDetails>) parms[1];
var adapter = new DetailsAdapter();
//we keep running until there are no records left inthe Database
while (!_noRecordsInPool)
{
foreach (var detail in details)
{
var userAdapter = new UserAdapter();
var domainAdapter = new DomainAdapter();
var user = userAdapter.FindById(detail.UserId);
var domain = domainAdapter.FindById(detail.DomainId);
//...do some work here......
adapter.Update(detail);
}
if (!_noRecordsInPool)
{
details = adapter.FindTopDetailsInQueue(_recordsPerthread);
if (details == null || details.Count <= 0)
{
_noRecordsInPool = true;
break;
}
}
}
}
The app crashes because there seem to be connection issues to the database. Looking in my log files for the DAL I am seeing this:
Timeout expired. The timeout period
elapsed prior to obtaining a
connection from the pool. This may
have occurred because all pooled
connections were in use and max pool
size was reached
When I run this in one thread it works fine. I am guessing when I runt his in multiple threads I am obviously making too many connections to the DB. Any thoughts on how I can keep this running in multiple threads and make sure the database doesn’t give me any errors.
Update:
I am thinking my issues may be deadlocks in my database. Here is the code in SQL that is running whe I get a deadlock error:
WITH cte AS (
SELECT TOP (#topCount) *
FROM
dbo.UserDetails WITH (READPAST)
WHERE
dbo.UserDetails where IsLocked = 0)
UPDATE cte
SET
IsLocked = 1
OUTPUT INSERTED.*;
I have never had issues with this code before (in other applications). I reorganzied my Indexes as they were 99% fragmented. That didn't help. I am at a loss here.
I'm confused as to where in your code connections get opened, but you probably want your data adapters to implement IDispose (making sure to close the pool connection as you leave using scope) and wrap your code in using blocks:
using (adapter = new UserDetailsAdapter())
{
for (var i = 1; i <= _threadCount; i++)
{
[..]
}
} // adapter leaves scope here; connection is implicitly marked as no longer necessary
ADO.NET uses connection pooling, so there's no need to (and it can be counter-productive to) explicitly open and close connections.
It is not clear to me how you actually connect to the database. The adapter must reference a connection.
How do you actually initialize that connection?
If you use a new adapter for each thread, you must use a new connection for each adapter.
I am not too familiar with your environment, but I am certain that you really need a lot of open connections before your DB starts complaining about it!
Well, after doing some research I found that there might be a bug in SQL server 2008 and running parallel queries. I’ll have to dig up the link where I found the discussion on this, but I ended up running this on my server:
sp_configure 'max degree of parallelism', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
This can decrease your server performance, overall, so it may not be an option for some people, but it worked great for me.
For some queries I added the MAXDOP(n) (n being the number of processors to utilize) option so they can run more efficiently. It did help a bit.
Secondly, I found out that my DAL’s Dispose method was using the GC.Suppressfinalize method. So, my finally sections were not firing in my DAL properly and not closing out my connections.
Thanks to all who gave their input!