Develop Reference

c# find start date and end date based on a list of dates?

I have a database table with over 200K+ records and a column containing a Date (NOT NULL). I am struggling to do a GroupBy Date since the database is massive the query takes soooo long to process (like 1 minute or so).
My Theory:
Get the list of all records from that table
From that list find the end date and the start date (basically the oldest date and the newest)
Then taking say like 20 dates to do the GroupBy on so the query will be done in a shorter set of records..
Here is my Model that I have to get the list:
registration.Select(c => new RegistrationViewModel()
{
DateReference = c.DateReference,
MinuteWorked = c.MinuteWorked,
});
The DateReferenceis the database column that I have to work with...
I am not pretty sure how to cycle through my list getting the dates start and end without taking too long.
Any idea on how to do that?
EDIT:
var registrationList = await context.Registration
.Where(c => c.Status == StatusRegistration.Active) // getting all active registrations
.ToRegistrationViewModel() // this is simply a select method
.OrderBy(d => d.DateReference.Date) // this takes long
.ToListAsync();
The GroupBy:
var grpList = registrationList.GroupBy(x => x.DateReference.Date).ToList();
var tempList = new List<List<RegistrationViewModel>>();
foreach (var item in grpList)
{
var selList = item.Select(c => new RegistrationViewModel()
{
RegistrationId = c.RegistrationId,
DateReference = c.DateReference,
MinuteWorked = c.MinuteWorked,
}).ToList();
tempList.Add(selList);
}
This is my SQL table:
This is the ToRegistrationViewModel() function:
return registration.Select(c => new RegistrationViewModel()
{
RegistrationId = c.RegistrationId,
PeopleId = c.PeopleId,
DateReference = c.DateReference,
DateChange = c.DateChange,
UserRef = c.UserRef,
CommissionId = c.CommissionId,
ActivityId = c.ActivityId,
MinuteWorked = c.MinuteWorked,
Activity = new ActivityViewModel()
{
Code = c.Activity.Code,
Description = c.Activity.Description,
},
Commission = new CommissionViewModel()
{
Code = c.Commission.Code,
Description = c.Commission.Description
},
People = new PeopleViewModel()
{
UserId = c.People.UserId,
Code = c.People.Code,
Name = c.People.Name,
Surname = c.People.Surname,
Active = c.People.Active
}
});

There are multiple potential problems here
Lack of indexes
Your query uses the Status and DateReference, and neither looks to have an index. If there are only a few active statuses a index on that column might suffice, otherwise you need a index on the date to speedup sorting. You might also consider a composite index that includes both columns. An appropriate index should solve the sorting issue.
Materializing the query
ToListAsync will trigger the execution of the sql query, making every subsequent operation run on the client. I would also be highly suspicious of ToRegistrationViewModel, I would try changing this to an anonymous type, and only convert to an actual type after the query has been materialized. Running things like sorting and grouping on the client is generally considered a bad idea, but you need to consider where the actual bottleneck is, optimizing the grouping will not help if the transfer of data takes most time.
Transferring data
Fetching a large number of rows will be slow, no matter what. The goal is usually to do as much filtering in the database as possible so you do not need to fetch so many rows. If you have to fetch a large amount of records you might use Pagination, i.e. combine OrderBy with Skip and Take to fetch smaller chunks of data. This will not save time overall, but can allow for things like progress and showing data continuously.

Replace view SQL Server with public List<> in c#.net

I am building a c#.net web application which draws data from a SQL Server. I have a view in SQL Server based on a single table containing multiple records for the employees. At most one record can be active, but it is not necessarily the one with MAX(ID). It contains the history as well as the current status of the employees. This is some legacy I have to work with.
In order to get the correct record I group by the employee code, and total the boolean field InDienst ('currently employed'), which can be 1 at most. Based on this selection I can go back and select the correct record applying MAX(ID).
SELECT
Personeelscode_ref AS Personeelscode,
SUM(InDienst) AS InDienst
FROM dbo.tPersoneel
GROUP BY Personeelscode_ref
This works fine.
However, as I am not the only one with access to the database, I would like to move this to the c#.net controller (or a model) and apply a List<> statement in c# which replaces the view. Something like this:
public static List<tPersoneel> listEmployeeAll = EmployeeDB.tPersoneel
.GroupBy(x => x.Personeelscode_ref)
.Select( new tPersoneel
{
Personeelscode_ref as Personeelscode,
(InDienst).sum()
})
.ToList();
Can you show me what the correct c#.net code would be?
Thanks much in advance!

use this code
List<tPersoneel> listEmployeeAll = EmployeeDB.tPersoneel.GroupBy(x => x.Personeelscode_ref)
.Select(y => new {
Personeelscode = y.Key.Personeelscode_ref,
InDienst = y.Sum(x => x.InDienst)
}).ToList();
or
List<tPersoneel> listEmployeeAll = (from c in EmployeeDB.tPersoneel
group c by c.Personeelscode_ref into g
select new
{
Personeelscode = y.Key.Personeelscode_ref,
InDienst = y.Sum(x => x.InDienst)
}).ToList();

EF6 IN clause on specific properties

Good morning,
I'm having trouble with a EF query. This is what i am trying to do.
First i am pulling a list of ID's like so (List of IDs are found in the included x.MappingAccts entity):
Entities.DB1.Mapping mapping = null;
using (var db = new Entities.DB1.DB1Conn())
{
mapping = db.Mappings.Where(x => x.Code == code).Include(x => x.MappingAccts).FirstOrDefault();
}
Later, i'm trying to do a query on a different DB against the list of Id's i pulled above (essentially a IN clause):
using (var db = new Entities.DB2.DB2Conn())
{
var accounts = db.Accounts.Where(mapping.MappingAccts.Any(y => y.Id == ?????????)).ToList();
}
As you can see i only got part way with this.
Basically what i need to do is query the Accounts table against it's ID column and pull all records that match mapping.MappingAccts.Id column.
Most of the examples i am finding explain nicely how to do this against a single dimension array but i'm looking to compare specific columns.
Any assist would be awesome.
Nugs

An IN clause is generated using a IEnumerable.Contains.
From the first DB1 context, materialize the list of Id's
var idList = mapping.MappingAccts.Select(m => m.Id).ToList();
Then in the second context query against the materialized list of id's
var accounts = db.Accounts
.Where(a => idList.Contains(a.Id))
.ToList();
The only problem you may have is with the amount of id's you are getting in the first list. You may hit a limit with the SQL query.

This will give the list of Accounts which have the Ids contained by MappingAccts
using (var db = new Entities.DB2.DB2Conn())
{
var accounts = db.Accounts.Where(s => mapping.MappingAccts.Any(y => y.Id == s.Id)).ToList();
}

How do you force a .Contains in a Linq to SQL where clause to be done by SQL server?

I have a list of row IDs being used in a Linq to SQL query.
using (var db = new DataContext(dbConnectionString))
{
ids = new list<long> {"1","2",...};
var data = (from item in db.GetTable<dataTable>().AsEnumerable()
where ids.Contains(item.ID)
select new customDataStructure{}).ToList();
}
In some cases, it passes each of the IDs in the list to SQL as parameters in the format "where ID in (#p0,#p1,...)". In other cases, the query retrieves all records and lets the filter happen in c#. This causes a huge spike in memory usage and possible memory exceptions it the amount of data retrieved is very large.
Is it possible to force the query to execute the where on the SQL server to avoid this issue?

Using .Contains() is not what I think you are wanting to use. I think you want to use .Any() which will see if any of the items in your list match a property or column in your table. See below:
var ids = new List<long> {1,2};
var data = (from item in db.GetTable<dataTable>().AsEnumerable()
where ids.Any(m=> m == item.Property)
select new customDataStructure{}).ToList();

As Evk commented, the AsEnumerable() caused the where to be executed locally, not on the SQL server.
It was being used since I needed lists and dictionaries in the returned type which cause exceptions if you try to build them in an IQueryable.
I altered the query to
using (var db = new DataContext(dbConnectionString))
{
ids = new list<long> {"1","2",...};
var data = (from item in db.GetTable<dataTable>()
where ids.Contains(item.ID)
select new
{
name = item.name,
subIds = item.subitemIDs
...
}).AsEnumerable()
.Select(x=> new customDataStructure
{
itemname = x.name,
subIds = x.subIds.ToList(),
...
}).ToList();
}
It let the data collection be done by SQL and then have the needed structure completed after the SQL executes.

Linq to SQL DataContext: how to load data?

(I'm completely new to Linq to SQL) I am creating a web app that works very closely with a database, I'm looking for the quickest and connection time efficient model and believing Linq to SQL to be this. I'm using C#/.Net4/Visual Studio 2010
For simplicity sake, I have a web .aspx page containing a number of asp Text Boxes. I want to give their Text values from SQL data via Linq to SQL object. I also have a file called DataClasses.dbml with a a table added in the design view. The code I have so far in my web page code-behind is:
DataClassesDataContext db = new DataClassesDataContext(getConnectionString);
var table = from t in db.MyTable
where t.PK == 2
select new { t.col1, t.col2, t.col3};
db.Connection.Open();
db. // What's the best way of loading the object?
db.Connection.Close();
How do I then access the column values? Or do I databind it to a datatable? If so, how?
myTextBox1.Text = table.???col1.value;

You don't need to open or close the connection. LinqToSql abstracts that away for you.
Once you created the query, you can execute it and retrieve the row as an object using SingleOrDefault().
using (var db = new DataClassesDataContext(getConnectionString))
{
var query = from t in db.MyTable
where t.PK == 2
select new { t.col1, t.col2, t.col3};
var myObject = query.SingleOrDefault();
}
You can also simplify this operation by using the lambda notation:
using (var db = new DataClassesDataContext(getConnectionString))
{
var myObject = db.MyTable.SingleOrDefault(t => t.PK == 2 )
}
To access the object you can directly access the columns since they have been mapped to the corresponding properties:
myTextBox1.Text = myObject.col1;

Common way is to call method that will execute query (ToArray, ToList, First, Single, etc..) or enumerate it in foreach.
For example:
var query = from t in db.MyTable
where t.PK == 2
select new { t.col1, t.col2, t.col3};
var result = query.ToArray(); // now it contains array of resulting objects
// or enumerate
foreach (var obj in query)
{
// do stuff with obj
}

Use
myTextBox1.Text = table.FirstOrDefault().col1.ToString();