I have an ADO.Net Data Access layer in my application that uses basic ADO.Net coupled with CRUD stored procedures (one per operation e.g. Select_myTable, Insert_myTable). As you can imagine, in a large system (like ours), the number of DB objects required by the DA layer is pretty large.
I've been looking at the possibility of refactoring the layer classes into EF POCO classes. I've managed to do this, but when I try to performance test, it gets pretty horrific. Using the class below (create object, set Key to desired value, call dataselect), 100000 runs of data loading only takes about 47 seconds (there are only a handful of records in the DB). Whereas the Stored Proc method takes about 7 seconds.
I'm looking for advice on how to optimise this - as a point of note, I cannot change the exposed functionality of the layer - only how it implements the methods (i.e. I can't pass responsibility for context ownership to the BO layer)
Thanks
public class DAContext : DbContext
{
public DAContext(DbConnection connection, DbTransaction trans)
: base(connection, false)
{
this.Database.UseTransaction(trans);
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
//Stop Pluralising the Object names for table names.
modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
//Set any property ending in "Key" as a key type.
modelBuilder.Properties().Where(prop => prop.Name.ToLower().EndsWith("key")).Configure(config => config.IsKey());
}
public DbSet<MyTable> MyTable{ get; set; }
}
public class MyTable : DataAccessBase
{
#region Properties
public int MyTableKey { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public bool Active { get; set; }
public int CreatedBy { get; set; }
public DateTime CreatedDate { get; set; }
public int ModifiedBy { get; set; }
public DateTime ModifiedDate { get; set; }
#endregion
#region constructors
public MyTable()
{
//Set Default Values.
Active = true;
Name = string.Empty;
CreatedDate = DateTime.MinValue;
ModifiedDate = DateTime.MinValue;
}
#endregion
#region Methods
public override void DataSelect(System.Data.SqlClient.SqlConnection connection, System.Data.SqlClient.SqlTransaction transaction)
{
using (DAContext ctxt = new DAContext(connection, transaction))
{
var limitquery = from C in ctxt.MyTable
select C;
//TODO: Sort the Query
limitquery = FilterQuery(limitquery);
var limit = limitquery.FirstOrDefault();
if (limit != null)
{
this.Name = limit.Name;
this.Description = limit.Description;
this.Active = limit.Active;
this.CreatedBy = limit.CreatedBy;
this.CreatedDate = limit.CreatedDate;
this.ModifiedBy = limit.ModifiedBy;
this.ModifiedDate = limit.ModifiedDate;
}
else
{
throw new ObjectNotFoundException(string.Format("No MyTable with the specified Key ({0}) exists", this.MyTableKey));
}
}
}
private IQueryable<MyTable1> FilterQuery(IQueryable<MyTable1> limitQuery)
{
if (MyTableKey > 0) limitQuery = limitQuery.Where(C => C.MyTableKey == MyTableKey);
if (!string.IsNullOrEmpty(Name)) limitQuery = limitQuery.Where(C => C.Name == Name);
if (!string.IsNullOrEmpty(Description)) limitQuery = limitQuery.Where(C => C.Description == Description);
if (Active) limitQuery = limitQuery.Where(C => C.Active == true);
if (CreatedBy > 0) limitQuery = limitQuery.Where(C => C.CreatedBy == CreatedBy);
if (ModifiedBy > 0) limitQuery = limitQuery.Where(C => C.ModifiedBy == ModifiedBy);
if (CreatedDate > DateTime.MinValue) limitQuery = limitQuery.Where(C => C.CreatedDate == CreatedDate);
if (ModifiedDate > DateTime.MinValue) limitQuery = limitQuery.Where(C => C.ModifiedDate == ModifiedDate);
return limitQuery;
}
#endregion
}
Selects are slow with tracking on. You should definitely turn off tracking and measure again.
Take a look at my benchmarks
http://netpl.blogspot.com/2013/05/yet-another-orm-micro-benchmark-part-23_15.html
This might be just a hunch, but ... In your stored procedure, the filters are well defined and the SP is in a compiled state with decent execution plan. Your EF query gets constructed from scratch and recompiled on every use. So the task now becomes to devise a way to compile and preserve your EF queries, between uses. One way would be to rewrite your FilterQuery to not rely on fluent conditional method chain. Instead of appending, or not, a new condition every time your parameter set changes, convert it into one, where the filter is either applied when condition is met, or overridden by something like 1.Equals(1) when not. This way your query can be complied and made available for re-use. The backing SQL will look funky, but execution times should improve. Alternatively you could devise Aspect Oriented Programming approach, where compiled queries would be re-used based on parameter values. If I will have the time, I will post a sample on Code Project.
Related
Lets say I have a domain model of an assembly line that has different orders on it. The user can change the value of the order but each order's value must be greater than the one in front of it and less than the one behind it. I have created an aggregate root called Line to enforce this invariant. This is the simplified version of that code below.
public class Line : IAggregateRoot
{
public int Id { get; }
public List<Order> Orders { get; }
public Line(int id, List<Order> orders)
{
Id = id;
Orders = orders;
}
public void SetOrderValue(int orderId, int newOrderValue)
{
var orderPos = Orders.FindIndex(o => o.Id == orderId);
if (orderPos != -1 && IsValidOrderValue(orderPos,newOrderValue))
{
Orders[orderPos].Value = newOrderValue;
}
}
private bool IsValidOrderValue(int orderPos, int newOrderValue)
{
var lessThanAfter = orderPos == Orders.Count - 1 ? true : newOrderValue <= Orders[orderPos + 1].Value;
var greaterThanBefore = orderPos == 0 ? true : newOrderValue <= Orders[orderPos - 1].Value;
return lessThanAfter && greaterThanBefore;
}
}
public class Order : IEntity<int>
{
public int Id { get; }
public int Value { get; set; }
/*
* Other info about the order goes here
*/
public Order(int id, int value)
{
Id = id;
Value = value;
}
}
The issue that I have is that any object that references Line can also change the value of any order and break the invariant.
line.Orders[0].Value = 10;
I know that in DDD, the aggregate root shouldn't allow references to the inner entities so I thought about making the orders list private. However, then when I try to store the Line aggregate root in a repository, the repository has no way of being able to fetch and save the list of orders. Is there a recommended way in DDD to protect the Order objects from outside objects being able to change their values while at the same time keeping the Order info public so the repository can save it in the database?
How to check if there exists an entry in the database? Debugging shows that the result returns null.
TimeSlots a Collection. I'm not sure if I've correctly done it.
Here is my context:
var result = await context.Bookings
.SingleOrDefaultAsync(b =>
b.BookDate == booking.BookDate
&& b.TimeSlots == booking.TimeSlots
&& b.RoomId == booking.RoomId);
public class Booking
{
public int Id { get; set; }
[Required]
public DateTime BookDate { get; set; }
[Required]
public int RoomId { get; set; }
public Room Room { get; set; }
[Required]
public ICollection<BookingTimeSlot> TimeSlots { get; set; }
[Required]
public ICollection<BookingModule> Modules { get; set; }
public Booking()
{
TimeSlots = new Collection<BookingTimeSlot>();
Modules = new Collection<BookingModule>();
}
}
public class BookingTimeSlot
{
public int BookingId { get; set; }
public int TimeSlotId { get; set; }
public Booking Booking { get; set; }
public TimeSlot TimeSlot { get; set; }
}
This is the input I'm trying to make:
{
"RoomId": 1,
"BookDate": "2020-10-27",
"TimeSlots": [1, 3],
"Modules": [1]
}
Your question is not clear whether you are looking to find duplicates or get the first result.
Second I think you are missing the primary key in your model or are you using a composite key, either way its good read to help you fix that.
//With Linq and EF find and process duplicates
// assuming Id is your primary key - [please fix this, some info for you][1]
var duplicateBookings = context.Bookings.GroupBy(i => i.id)
.Where(x => x.Count() > 1)
.Select(val => val.Key); // or .SelectMany(i => i.ToList());
// do what you need
foreach(var dupes in duplicateBookings )
{
//process or do what you need
context.Bookings.DeleteObject(dupes); // for e.g. delete duplicate bookings
}
If you just want the first result, then change the sing to first
var result = await context.Bookings
.FirstOrDefaultAsync(b => //first result
b.BookDate == booking.BookDate
&& b.TimeSlots == booking.TimeSlots
&& b.RoomId == booking.RoomId);
Well, for the most part referential integrity in the database does not solve such problems, and in the vast majority of cases RI is not used anyway. The reason is that's not the job of RI but worse such RI violations tend to occur FAR TOO late for a user interface that informs the user that such a booking cannot be done. In other words you don't try to make the booking, keep fingers crossed, and the hope the data can be written out.
WHAT you do is provide a UI that when the user selects a booking date, you give feedback that such a booking can't be made, and as such NO DATABASE writes or updates will have YET occurred - hence this is a UI issue, not really a database RI issue. And even if it was a database RI issue, you would have to attempt to write the data, and often the user is JUST checking and asking for a particular booking date - not necessary ready to actually book.
a booking collision can be found based on this logic:
RequestStartDate <= EndDate
and
RequestEndDate >= StartDate
So any overlap or even a full bracketing will be found with the above simple query.
So, with above? Then with a room number and list of booking date ranges for that room, then a collision would be found with this:
#dtRequestStartDate = "Enter start Date"
#dtRequestEndDate = "Enter end date"
#RoomNum = Room number
strSQL = SELECT * from tblBookings where
(#dtRequestStartDate <= RoomEndDate)
AND
(#dtRequestEndDate >= RoomStartDate)
AND
(#RoomNum = RoomNumber)
If above row.Count > 0 then
message = Sorry, you cannot book that room
So what you do is check before a booking, and if above returns rows, then you don't allow the booking. As long as you never allow overlaps for bookings, then the above SIMPLE logic will always work and always prevent a booking with collisions.
I have found a solution to my problem. I compare the TimeSlotId using Intersect, if there are intersection, it then return true.
public bool BookingExist(Booking booking)
{
var resultContext = context.Bookings
.Where(b => b.Room.Id == booking.RoomId && b.BookDate == booking.BookDate)
.SelectMany(b => b.TimeSlots.Select(bt => bt.TimeSlotId))
.AsEnumerable();
var resultInput = booking.TimeSlots.Select(bt => bt.TimeSlotId);
if (resultContext.Intersect(resultInput).Count() > 0)
return true;
else
return false;
}
I'm working on a project that was not designed with unit testing in mind.
Since inside of StartWorking() method I create a new instance of WorkYear and call year.RecalculateAllTime(), is WorkYear class considered to be an external dependency (in terms of unit testing)?
Or since the Employee bound to WorkYear by composition relationship + Employee is meant to perform actions on WorkYears, are the Employee and WorkYear form a cohesive entity where both classes aren't considered as dependencies of one another?
In other words, should StartWorking(...) method be tested in isolation from WorkYear class?
public abstract class Employee
{
private List<WorkYear> _workYears;
private readonly IntervalCalculator _intervalCalculator;
// Other fields...
protected Employee(IntervalCalculator intervalCalculator)
{
_intervalCalculator = intervalCalculator;
WorkYears = new List<WorkYear>();
}
public IEnumerable<WorkYear> WorkYears
{
get => _workYears.AsReadOnly();
private set => _workYears = value.ToList();
}
// Other properties...
public void StartWorking(DateTime joinedCompany)
{
List<PayPeriodInterval> allIntervals = _intervalCalculator.GenerateIntervalsFor(joinedCompany.Date.Year);
PayPeriodInterval currentInterval = allIntervals.Find(i => i.StartDate <= joinedCompany && joinedCompany <= i.EndDate);
PayPeriod firstPeriod = CalculateFirstPeriod(joinedCompany, currentInterval);
// There is a possibility that employee worked during this year and returned during
// the same exact year or even month. That is why we are trying to find this year in database:
WorkYear year = WorkYears.FirstOrDefault(y => y.CurrentYear == joinedCompany.Year);
if (year == null)
{
// Create new year with current and future periods.
year = new WorkYear(joinedCompany.Year, this, new List<PayPeriod> {firstPeriod});
AddYear(year);
}
else
{
// There is a possibility that employee left and got back during the same period.
// That is why we should try to find this period so that we don't override it with new one:
PayPeriod existingPeriod = year.GetPeriodByDate(joinedCompany);
if (existingPeriod != null)
{
var oldCurrentPeriodWorktime = new TimeSpan(existingPeriod.WorktimeHours, existingPeriod.WorktimeMinutes, 0);
firstPeriod = CalculateFirstPeriod(joinedCompany, currentInterval, oldCurrentPeriodWorktime);
}
year.PayPeriods.Add(firstPeriod);
}
List<PayPeriodInterval> futureIntervals = allIntervals.FindAll(i => currentInterval.EndDate < i.StartDate);
List<PayPeriod> futurePeriods = NewPeriods(futureIntervals);
year.PayPeriods.AddRange(futurePeriods);
year.RecalculateAllTime();
}
public abstract List<PayPeriod> NewPeriods(List<PayPeriodInterval> intervals);
public void AddYear(WorkYear workYear) => _workYears.Add(workYear);
protected abstract PayPeriod CalculateFirstPeriod(DateTime firstDayAtWork, PayPeriodInterval firstPeriodInerval, TimeSpan initialTime = default);
// Other methods...
}
public class WorkYear
{
public WorkYear(int currentYear, Employee employee, List<PayPeriod> periods)
{
Employee = employee;
EmployeeId = employee.Id;
CurrentYear = currentYear;
PayPeriods = periods ?? new List<PayPeriod>();
foreach (PayPeriod period in PayPeriods)
{
period.WorkYear = this;
period.WorkYearId = Id;
}
}
public int EmployeeId { get; }
public int CurrentYear { get; }
public Employee Employee { get; }
public List<PayPeriod> PayPeriods { get; set; }
// Other roperties...
public void RecalculateAllTime()
{
//Implementation Logic
}
}
Super big caveat: This stuff gets really opinionated really fast. There are lots of valid designs!
Okay, now that I've said that. DTO's (Data transfer objects) are not external dependencies. You don't inject them.
WorkYear has all the hallmarks of a DTO (other than that method). I think you are OK as is. Because it has that RecalculateAllTime method it should also be unit tested however. An example of an external dependency in this case would be something that fetches the list of work years.
The basic rule of thumb is:
You compose data (DTOs)
You inject behavior (services)
I like LinqToSQL because it is waaaayyy simpler to implement in a small project (versus EF) and avoids me to write/maintain SQL.
I have a lot of working Entity/tables but I face a problem with a particular class that use inheritance.
The SQL table "ChildClass" have only 3 fields : id, MyOtherField01 and MyOtherField02. A primary key is created on "id".
here is the class [simplified] :
[Table("ChildClass")]
public class ChildClass : ParentClass {
//constructor
public ChildClass(){}
//constructor that take the parent to inject property
public ChildClass(ParentClass ParamInject)
{ //code removed for simplicity
}
[Column(IsDbGenerated = true, Name = "id", IsPrimaryKey = true)]
public int NoUnique { get; set; }
[Column]
public bool MyOtherField01 { get; set; }
[Column]
public DateTime? MyOtherField02 { get; set; } }
}
Because of many reason, here is my insert/update mecanism that is able to ignore the context :
var builder = new StringBuilder();
try
{
using (var MyDBConnexion = new SqlConnection(MyDBConnStr))
{
//ouverture du context
using (BusinessContext MyDBContext = new BusinessContext(MyDBConnexion))
{
MyDBContext.Log = new System.IO.StringWriter(builder);
//If we have an insert...
if (param_Client.NoUnique == 0)
MyDBContext.ListOfChildClasses.InsertOnSubmit(MyChildClassInstance);
else //if we must update
{
//little funky work around that normally works well, that allow me to dont care about Context
var Existing = MyDBContext.ListOfChildClasses.Single(x => x.NoUnique == MyChildClassInstance.NoUnique);
//than, copy the properties into the "existing"
param_Client.CopyProperties(Existing);
}
MyDBContext.SubmitChanges();
}
}
}
catch (Exception ex)
{
string strT = builder.ToString(); //Here, I have the Update statement!
return strErr;
}
return "";
But for this precise class, here is the resulting SQL :
UPDATE ChildClass
SET
WHERE (id = #p1 and PrivilegeActif = #p2 and DateAdhesion = #p3)
No field in the set section!
any ideas?
We're currently trying to use the Task<IAsyncEnumerator<StreamResult<T>>> StreamAsync<T>(IQueryable<T> query, CancellationToken token = null), running into some issues.
Our document look something like:
public class Entity
{
public string Id { get; set; }
public DateTime Created { get; set; }
public Geolocation Geolocation { get; set; }
public string Description { get; set; }
public IList<string> SubEntities { get; set; }
public Entity()
{
this.Id = Guid.NewGuid().ToString();
this.Created = DateTime.UtcNow;
}
}
In combination we've a view model, which is also the model were indexing:
public class EntityViewModel
{
public string Id { get; set; }
public DateTime Created { get; set; }
public Geolocation Geolocation { get; set; }
public string Description { get; set; }
public IList<SubEntity> SubEntities { get; set; }
}
And ofcourse, the index, with the resulttype inheriting from the viewmodel, to enable that SubEntities are mapped and output correctly, while enabling the addition of searchfeatures such as fulltext etc.:
public class EntityWithSubentitiesIndex : AbstractIndexCreationTask<Entity, EntityWithSubentitiesIndex.Result>
{
public class Result : EntityViewModel
{
public string Fulltext { get; set; }
}
public EntityWithSubentitiesIndex ()
{
Map = entities => from entity in entities
select new
{
Id = entity.Id,
Created = entity.Created,
Geolocation = entity.Geolocation,
SubEntities = entity.SubEntities.Select(x => LoadDocument<SubEntity>(x)),
Fulltext = new[]
{
entity.Description
}.Concat(entity.SubEntities.Select(x => LoadDocument<SubEntity>(x).Name)),
__ = SpatialGenerate("__geolokation", entity.Geolocation.Lat, entity.Geolocation.Lon)
};
Index(x => x.Created.Date, FieldIndexing.Analyzed);
Index(x => x.Fulltext, FieldIndexing.Analyzed);
Spatial("__geolokation", x => x.Cartesian.BoundingBoxIndex());
}
}
Finally we're querying like this:
var query = _ravenSession.Query<EntityWithSubentitiesIndex.Result, EntityWithSubentitiesIndex>()
.Customize(c =>
{
if (filter.Boundary == null) return;
var wkt = filter.Boundary.GenerateWkt().Result;
if (!string.IsNullOrWhiteSpace(wkt))
{
c.RelatesToShape("__geolokation", wkt, SpatialRelation.Within);
}
})
.AsQueryable();
// (...) and several other filters here, removed for clarity
var enumerator = await _ravenSession.Advanced.StreamAsync(query);
var list = new List<EntityViewModel>();
while (await enumerator.MoveNextAsync())
{
list.Add(enumerator.Current.Document);
}
When doing so we're getting the following exception:
System.InvalidOperationException: The query results type is 'Entity'
but you expected to get results of type 'Result'. If you want to
return a projection, you should use
.ProjectFromIndexFieldsInto() (for Query) or
.SelectFields() (for DocumentQuery) before calling to
.ToList().
According to the documentation, the Streaming API should support streaming via an index, and querying via an IQueryable at once.
How can this be fixed, while still using an index, and the streaming API, to:
Prevent having to page through the normal query, to work around the default pagesize
Prevent having to load the subentities one at a time when querying
Thanks in advance!
Try to use:
.As<Entity>()
(or .OfType<Entity>()) in your query. That should work in the regular stream.
This is a simple streaming query using "TestIndex" that is an index over an entity Test and I'm using a TestIndex.Result to look like your query. Note that this is actually not what the query will return, it's only there so you can write typed queries (ie. .Where(x => x.SomethingMapped == something))
var queryable = session.Query<TestIndex.Result, TestIndex>()
.Customize(c =>
{
//do stuff
})
.As<Test>();
var enumerator = session.Advanced.Stream(queryable);
while (enumerator.MoveNext())
{
var entity = enumerator.Current.Document;
}
If you instead want to retrieve the values from the index and not the actual entity being indexed you have to store those as fields and then project them into a "view model" that matches your mapped properties. This can be done by using .ProjectFromIndexFieldsInto<T>() in your query. All the stored fields from the index will be mapped to the model you specify.
Hope this helps (and makes sense)!
Edit: Updated with a, for me, working example of the Streaming API used with ProjectFromIndexFieldsInto<T>() that returns more than 128 records.
using (var session = store.OpenAsyncSession())
{
var queryable = session.Query<Customers_ByName.QueryModel, Customers_ByName>()
.Customize(c =>
{
//just to do some customization to look more like OP's query
c.RandomOrdering();
})
.ProjectFromIndexFieldsInto<CustomerViewModel>();
var enumerator = await session.Advanced.StreamAsync(queryable);
var customerViewModels = new List<CustomerViewModel>();
while (await enumerator.MoveNextAsync())
{
customerViewModels.Add(enumerator.Current.Document);
}
Console.WriteLine(customerViewModels.Count); //in my case 504
}
The above code works great for me. The index has one property mapped (name) and that property is stored. This is running the latest stable build (3.0.3800).
As #nicolai-heilbuth stated in the comments to #jens-pettersson's answer, it seems to be a bug in the RavenDB client libraries from version 3 onwards.
Bug report filed here: http://issues.hibernatingrhinos.com/issue/RavenDB-3916