Linq not finding matching record that is there - c#

I'm trying to find a record that I know exist in the database. When searching for it using this query nothing is found.
List<OrganizationALE> ales =
_ctx.OrganizationALEs.Where(c => c.OrganizationId.Equals(organizationId) && c.LastModified.Equals(modified) && c.StartDate.Equals(start)).ToList();
But when I search using this query it is clearly there in the result using watch.
And if I try pull it out on the very next line using
var found = ales.First(a => a.LastModified == modified);
I get an exception say Sequence contains no matching element

I bet it's because modified and/or start contains milliseconds that are not stored with the same precision in the database (datetime in SQL Server?) so that the comparison fails in the database.
Possible solutions are here (the second and third point apply to your situation). Basically supply less precise DateTime values (cut off the millisecond) or use datetime2(7) in SQL Server or avoid using equality comparisons with DateTime values and use >= and <= instead.

Related

LINQ query maps dates incorrectly

My query looks like so:
using (var ctx = new PCLvsCompContext())
{
var brokerId = broker?.Id;
var symbolId = company?.Id;
var result = (from t in ctx.TradeHistoryJoineds
where t.TradeDate >= fromDate
&& t.TradeDate <= toDate
&& (brokerId == null || t.BrokerId == brokerId)
&& (symbolId == null || t.SymbolId == symbolId)
select t).OrderBy(x => x.TradeDate).ThenBy(x => x.BrokerName).ToList();
return result;
}
As an example, I run this query with dates like fromDate March-01-2017 toDate March-31-2017. I then captured the generated sql in SQL profiler that this query produces and ran it in SQL management studio. The output was as expected where for each weekday, each company has some trades. The query is based off of a view which casts all dates to "datetime" so that excel can parse them as dates correctly. However, when I put a breakpoint at "return result" and inspect the dates, all but 2 of the dates are March-1-2017. This is incorrect, the query result in SQL manager shows trades for almost every weekday in March (which is correct).
What is going on here? Why is Linq losing its mind?
Although based on the results I cannot see exactly how you would end up with those results, it is very common that you could be dealing with a DateTime timezone issue. I suspect that perhaps you saved your dates to the database using a DateTime object from say DateTime.Now instead of DateTime.UtcNow. So at that point in time and based on the machine it was called on it would be based on the timezone and datelight savings of that machine.
A DateTime object should not be used as it can relate to the region of the SQL database, the region of the server making this LINQ call and so the two regions could be on different timezones.
Instead you should always use DateTimeOffset.
If you cannot do that for some reason, then double-check your dates toDate and fromDate and do:
var utcToDate = toDate.ToUniversalTime().ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'");
var utcFromDate = toDate.ToUniversalTime().ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'");
Which gives something like this if it was run on 3rd April 2018 at 22:56.
2018-04-03T22:56:57.740Z
You would then also need to make sure when you save any date to the SQL backing store that you do ToUniversalTime() firstly. You can check your SQL tables directly with a normal SQL query and they should be stored in the database as the same UTC string format as above, and then the comparison should be obvious to whether it is valid.
However I would strongly recommend changing all your DateTime calls and gets to DateTimeOffset. The only time to use DateTime in majority of cases is for the final display to a user.
Thank you all for your suggestions. For those who are familiar with linq, EF and views this may seem like a stupid oversight, but I will post my shame for others in case this happens to them since the cause and the resulting behavior are not immediately obviously linked (as can be seen by all the suggestions, none of which actually point to the answer).
When you are querying a view using linq and Entity Framework, you apparently must have a column called 'Id', otherwise Entity Framework can't distinguish any of the rows from one another and simply makes some sort of duplication that I can't quite decipher (all the rows were unique based on their data even without an Id column, so I can't quite figure out why this is necessary).
So by adding an the TradeId with an 'Id' alias to the view, then Entity Framework seemed to return to sanity and map the data as expected.

LINQ to SQL query succeeds or fails based on using foreign key versus using a built in navigation property

I have a query against a SQL server database that throws an Error converting data type nvarchar to numeric. exception. I was trying to use Convert.ToDecimal on a varchar field, however I'll eyeballed the data as best I could and couldn't find an invalid value.
The query is filtering the table by a 'Group' by using the foreign key of p.pgKey=#. However, if I use the navigation property and filter by the navigation property instead, p.Group.gName='ABC' the query works.
Here are the queries (note, originally, I didn't know if error was happening in Where translation or the Select processing, so that is why query looks weird, but as you can guess, when it works, it should just return a single distinct row of true):
Profiles
.Where(p =>
p.pgKey == 237
&& !p.pPlanProfile.Value
&& Convert.ToDecimal(p.pSearch08 ?? "0") > 0
).Select(p =>
Convert.ToDecimal(p.pSearch08 ?? "0") > 0
)
.Distinct()
.Dump();
The above query fails, while this query succeeds:
Profiles
.Where(p =>
p.Groups.gName == "ABC"
&& !p.pPlanProfile.Value
&& Convert.ToDecimal(p.pSearch08 ?? "0") > 0
).Select(p =>
Convert.ToDecimal(p.pSearch08 ?? "0") > 0
)
.Distinct()
.Dump();
Below is a full LINQPad screen dump showing:
Proves the gKey for ABC is 237.
Proves that the counts of simplying counting the Profile records is identical when using pgKey or Group.gName.
Shows the query working correctly when using the Group.gName processing.
Shows the query failing when using the pgKey processing.
Obviously I've used the Group.gName method to fix my problem, but I stumbled on that solution by accident. Anyone know why LINQ to SQL would behave this way?
Note: I get the same behavior using the generated DataContext from LINQPad or if I run against a compiled .dbml DataContext.
The two queries will be generating different TSQL and hence the query plan will be different.
I would suspect that the former query is attempting to convert some values of pSearch08 to decimal before it rejects them based on the other selection criteria, whereas the latter query is performing the other selection criteria first and hence is attempting to convert a smaller number of pSearch08 values to decimal and hence does not attempt to convert the invalid value.
If this is the case, then it could be dangerous to assume that the second query will always work and it would be best to fix the invalid data.
Rather than eyeballing the data you can try
SELECT * from Profile where ISNUMERIC(pSearch08) = 0

How can I speed up my EF query?

How can I speed up this query? Right now it's taking me around 2 minutes to pull back 210K records.
I turned off LazyLoading as well as set AsNoTracking on my tables.
I know it's a lot of data but surely it shouldn't take 2 minutes to retrieve the data?
context.Configuration.LazyLoadingEnabled = false;
List<MY_DATA> data = context
.MY_DATA.AsNoTracking()
.Include(x => x.MY_DATA_DETAILS)
.Where(x => startDate <= DbFunctions.TruncateTime(x.DB_DATE)
&& endDate >= DbFunctions.TruncateTime(x.DB_DATE)
&& x.MY_DATA_DETAILS.CODE.Trim().ToUpper() == myCode.Trim().ToUpper())
.ToList();
You can do without DbFunctions.TruncateTime() and probably also without these Trim().ToUpper() calls.
If you execute a function on a database column before it is filtered, it's impossible for the query optimizer to use any indexes on this column. This is know as being non-sargable. To execute the query in its present form, the database engine has to transform the data first and then scan all the transformed data to do the filtering. No index involved.
DbFunctions.TruncateTime() is meaningless. You have to choose startDate and endDate wisely and use x.DB_DATE as it is.
Further, if x.MY_DATA_DETAILS.CODE is a varchar column (most text columns are), it will be auto-trimmed in searches. Even if the database value contains trailing spaces, they will be ignored. So Trim isn't necessary. Next, most text columns by default have a case-insensitive database collation. You should check it. If this is SQL Server, look for collations like SQL_Latin1_General_CP1_CI_AS. The CI part means Case-Insensitive. If so, you can also do away with the ToUpper part. If not, you can either change the collation to a case-insensitive one, or you maybe should conclude that the column is case sensitive for a reason, so it does matter whether you look for Abc or abc.
Either way, having these transforming function removed form the database columns, the query should be able to run considerably faster, provided that proper indexes are in place.
Usually defining indexes on the columns that you frequently use in the 'where' clause, can improve your performance in selecting the rows from a large tale.
I recommend that you create a stored procedure and move your query into the SP and apply the performance tuning in Database and in your C# code, call the SP.
In addition to what the other guy said about moving your query to a stored proc and creating the proper indexes, I'd say you'd be better off using SQL reporting rather then trying to import the data into your application and reporting from there. Especially 210K rows. SQL has internal optimizations that you'll never be able to come close to with stored procs and queries.
You can see this very easily:
1) try write a simple console app that tries to pull down that entire table and writes it to a CSV file -- it'll be extremely slow.
2) try use the data export through Sql Mgmt Studio and export to a CSV -- it'll be done in a few seconds.
Do you need all the attributes of the object?
You can do this.
List<MY_DATA> data = context.MY_DATA.Include(x =>
x.MY_DATA_DETAILS).Where(x => startDate <= DbFunctions.TruncateTime(x.DB_DATE) &&
endDate >= DbFunctions.TruncateTime(x.DB_DATE) &&
x.MY_DATA_DETAILS.CODE.Trim().ToUpper() == myCode.Trim().ToUpper()).select(x => new MY_DATA()
{
Value = data
}).ToList();

Compare String dates from mysql database with entity framework c#

I have a database field like this:
Timestamp varchar(19) utf8mb4_unicode_ci
containing a timestamp (string) like this
"2013-05-29 00:00:00"
I am using the entity framework and I would like to filter by time - meaning I would like to get all entries having a timestamp > (now-interval).
Code would look something like this
var query = from r in db.route
where
r.timestamp > (now-interval);
select r;
How can I do this?
My first suggestion would be to fix the database so the date values are stored as the correct date type. That would solve many issues as well as increase search performance. However, in the (unlikely) situation that you are unable to do that, and that the format of the values in that column all match exactly as you specified in the question, you could convert your local time stamp variable to a string and compare it directly. As the format you have shown has an alphanumeric ordering that is identical to the date order, it should work:
//It's important that this is done here and not inline with the Linq query
var nowInterval = timeStamp.ToString("yyyy-MM-dd HH:mm:ss");
var query = from r in db.route
where string.Compare(r.timestamp, nowInterval, StringComparison.Ordinal) > 0
select r;

Convert a datetime in a subcollection of collection and use it in LINQ to SQL

I have a collection with subcollections in it, one of which is a date field stored in the DB as a string in the format yyyymmdd (which also contains a few random things such as "E" or 20085, etc.). This date is also part of one of the subcollections in the collection. This date field will now be used for searching, so I need to make it into a real date to use it in LINQ statements.
I since learned that LINQ to SQL doesn't support statements that it cannot translate into SQL, so I can't insert a function that returns a properly converted date and I haven't yet found a standard convert function that will transform the string into a valid date object.
I also tried layered converting, though this smelled bad, such as this:
search = from c in search
where c.Object.Any(p=> new DateTime(Convert.ToInt32(p.theDate, Substring(0,4))))... etc.
I just received conversion errors no matter what I did here.
My basic question is: does LINQ to SQL support anything inline for such a conversion? Another option is to change the datamodel or make a view, but I'd rather see if there are ways to handle this in code first. What I'm trying to do is something like the following:
search = from c in search
where c.subcollection.Any(p=>p.theDate >= min) && c.subcollection.Any(p=>p.theDate <= max)
select c;
Where min and max are passed in date values.
Thank you!
The SqlMethods class has some static helper methods that are translatable by LINQ to SQL. Most of these have to do with DateTime comparison. I'm not sure exactly which method you are looking for, but you should be able to find what you want at the documentation page for System.Data.Linq.SqlClient.SqlMethods
Alternatively, you could create a user-defined function and handle the conversion and comparison issue on the SQL side. The integration doesn't require much more than decorating the corresponding Object Model method with a FunctionAttribute, (which indicates how LINQ-to-SQL should process the method. Here is some documentation on this process:
How to: Call User-Defined Functions Inline
User-Defined Functions
If you wish to compare a date using LINQ but the SQL date column is a string value, use .CompareTo("2002-01-01") >= 0
This will translate and return any dates that match 2002-01-01 or occured later in time. Not sure how this works in with your subcollection but as an example:
search = from c in search
where c.stringDateColumn.CompareTo("2002-01-01") >= 0
select c;
Edit: I did some tests on a table with dates in strings like "2002-09-04E", "2004-04-01 34", "20050604e" and "2005-06-01444" and the results do return based on a string in format yyyy-mm-dd or yyyymmdd:
search = from c in search
where c.subcollection.Any(p=>p.theDate.CompareTo(min) >= 0) && c.subcollection.Any(p=>p.theDate.CompareTo(max) <= 0)
select c;

Categories

Resources