Develop Reference

Fastest way of retrieving records from Entity Framework DbSet on varchar Unix Timestamp

I have to retrieve all the records from a database that have been added since the last execution, this should happen daily.
The only thing that can identify those records from the rest is a Unix Timestamp (in milliseconds) or a Time (hhmmss) and a Date (yyyyMMdd). My problem is that all these columns are of type varchar.
The database is very big, and only getting bigger. Is there any way of getting only the rows with a Unix Timestamp higher than X without having to load the entire thing and parsing the timestamp?
What I do now is:
var sales = context.SALES.Select(s =>
new Sale {
Product = s.SC_PRDCT,
Terminal = s.SC_TERM,
Operator = s.MC_OP,
UnixString = s.SC_TIMESTAMP
})
.ToList()
.Where(m => terminals.ContainsKey(m.Terminal) && m.UnixTime > lastExecution);
public string UnixString
{
get { return unixString; }
set { unixString = value; UnixTime = long.Parse(value); }
}

Options that come to mind: If you have the ability to alter the schema while preserving the current fields I would consider adding a computed column to the database for a DateTime equivalent to the timestamp. Barring that, using a View to source the data for this type of search/report which can provide the translated timestamp.
If you don't have the ability to adjust the schema, then things will get a bit trickier. When you say the timestamp can be milliseconds or datetime in a string, does that mean the value can be something like either "1435234353353" (ms since Date X) or "20190827151530" for 2019-08-27 3:15:30 PM? If that is the case, as long as the length of the 2 strings, however formatted, is different then you can potentially still query against the field, it just won't be ideal:
Assuming the date option formatting is "YYYYMMDDHHMMSS":
string lastExecutionDate = targetDate.ToString("yyyyMMddHHmmss");
string lastExecutionMs = targetDate.ToUniversalTime().Subtract(
new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
).TotalMilliseconds.ToString();
var terminalKeys = terminals.Keys.ToList(); // You probably cannot use ContainsKey in Linq2EF...
var sales = context.SALES
.Where(s => terminalKeys.Contains(s.SC_TERM)
&& ((s.SC_TIMESTAMP.Length == 14 && s.SC_TIMESTAMP.CompareTo(lastExecutionDate) > 0)
|| (s.SC_TIMESTAMP.Length != 14 && s.SC_TIMESTAMP.CompareTo(lastExecutionMs) > 0 )))
.Select(s =>
new Sale
{
Product = s.SC_PRDCT,
Terminal = s.SC_TERM,
Operator = s.MC_OP,
UnixString = s.SC_TIMESTAMP
}).ToList();
If the SC_TIMESTAMP column only stores the timestamps in ms, and the time/date are in separate columns, then you don't need the conditional, just format your target datetime to a timestamp string (ms since 1970-01-01) and use that.
string lastExecutionMs = targetDate.ToUniversalTime().Subtract(
new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
).TotalMilliseconds.ToString();
var terminalKeys = terminals.Keys.ToList(); // You probably cannot use ContainsKey in Linq2EF...
var sales = context.SALES
.Where(s => terminalKeys.Contains(s.SC_TERM)
&& s.SC_TIMESTAMP.CompareTo(lastExecutionMs) > 0)
.Select(s =>
new Sale
{
Product = s.SC_PRDCT,
Terminal = s.SC_TERM,
Operator = s.MC_OP,
UnixString = s.SC_TIMESTAMP
}).ToList();
The caveat for this to work with the ms or datetime in the same field is that if you require timestamp or datetime is that the datetime string must be an ISO format "year Month Day Time24" which is sortable, otherwise it cannot be used in a comparison.

The lucky fact that the the unix timestamps are all equal in length, thus sortable, you can query them by the > operator in SQL.
Of course, as you' may have tried, m.SC_TIMESTAMP > lastExecution doesn't compile in C# but fortunately, EF6 (and EF3, more or less) translate the following predicate into the desired SQL predicate:
Where(m => m.SC_TIMESTAMP.CompareTo(lastExecution) > 0)
where lastExecution is a string.
Remember to add a comment to your code that this works until 2286-11-20 17:46:39, when the UNIX timestamp is 9999999999999. After that, your successors should use a more generic method that takes length into account.

Storing DateTime in PostgreSQL without loosing precision [duplicate]

This question already has answers here:
What is the most elegant way to store timestamp with nanosec in postgresql?
(3 answers)
Closed 4 years ago.
I have stumped upon an issue, that whenever I load what is presumably a DateTime from Postgresql database and compare it to initial value from C# - then the data does not match. For example, here's an original DateTime in C#:
Date: {09.07.2018 00:00:00}
Ticks: 636667391123714378
Time: {13:18:32.3714378}
Same date, returned from DB (after it was saved):
Date: {09.07.2018 00:00:00}
Ticks: 636667391123714370
Time: {13:18:32.3714370}
It seems like upon save - DateTime has lost some precision. Column type in database is a Timestamp. So, 2 questions:
What will be a proper way of storing the DateTime in PostgreSQL, so I don't loose any precision?
I was thinking about saving a number of UNIX milliseconds into an Integer field instead, of saving DateTime as a Timestamp, i.e.:
DateTime myDT = DateTime.Now;
long ms = new DateTimeOffset(myDT).ToUnixTimeMilliseconds();
// how I do it now
var parameterOld =
new Npgsql.NpgsqlParameter("dateInserted", NpgsqlTypes.NpgsqlDbType.Timestamp) {Value = myDT };
// how I think would be a better approach
var parameterNew =
new Npgsql.NpgsqlParameter("dateInserted", NpgsqlTypes.NpgsqlDbType.Integer) { Value = ms };
Alternatively, how do I compare 2 datetimes with a highest possible precision (C#/PostgreSql)?
My current solution, which is kind of working.. But I am loosing too much precision here:
public static bool IsEqualWithLossOfPrecision(this DateTime timestamp1, DateTime timestamp2)
{
return (
timestamp1.Year == timestamp2.Year &&
timestamp1.Month == timestamp2.Month &&
timestamp1.Day == timestamp2.Day &&
timestamp1.Hour == timestamp2.Hour &&
timestamp1.Minute == timestamp2.Minute &&
timestamp1.Millisecond == timestamp2.Millisecond
);
}
Any advises on 2 questions are welcomed.
I am using the latest available version of DB (10.4), Npgsql library v. 4.0.0, .net framework 4.5ish, windows 10

PostgreSQL's timestamps have a precision limit of microseconds.
If you need more, store the nanoseconds in a separate bigint attribute.

list data in between from and to dates for current date [duplicate]

This question already has answers here:
C# Linq Where Date Between 2 Dates
(9 answers)
Closed 6 years ago.
I have a table with some date range assigned to user. i need to get details of user for whom current day is assigned.
user from-date to-date
------------------------------
x1 12/08/2016 25/08/2016
x2 24/08/2016 30/08/2016
x3 15/08/2016 20/08/2016
x4 18/08/2016 27/08/2016
the above given is table details
i want to list user, x1 and x4 to be listed as it is assigned to current date(22/08/2016)
thanks in advance

You need something like this:
var users = dbContext.Users.Where(user.FromDate < DateTime.Now
&& user.ToDate > DateTime.Now)
.Select(user=>user.User);
I have supposed that you have a context class through which you speak with your database and this context class has a connection to the Users table. Even if this is not the case, if you change accordingly the first part, dbContext.Users, the rest part is that you want (a few renamings may be needed)

A simple place to start with linq will be Where:
DateTime date = DateTime.Now;
var result = dbContext.table.Where(item => item.FromDate >= date && item.ToDate < date)
.Select(item => item.User);

return date without time but not of type string but date

I have a date value that I want to strip the time from. I want the return type to be a date type since I want to order the list of date I have. having a list to string representing Date does not return a correct order.
I know that DateTime always returns the date with the time. What are my options here? How can I better my code and have a list of items of Date type without the time?
Edit: I would like to have the date only. nothing after that. Something like 8/7/2016 not 8/7/2016 00:00:00 or anything after date. and in a date object.
Here is my code:
using (var db = new MyModel.Context())
{
var cert = (
from tr in db.uspTranscripts(personId)
from a in db.PersonTranscripts.Where(x => x.UPID == personId)
from b in db.LU_CreditType.Where(x => x.ID == a.CreditTypeID)
select new CertViewModel
{
ActivityTitle = tr.ActivityTitle,
Score = tr.Score,
Status = tr.Status,
CompletionDate = tr.CompletionDate,
CretitTypeName = b.ShortName,
CompletedDateSorted = a.HK_CreatedOn
}).OrderByDescending(x => x.CompletedDateSorted).ToList();
List<CertViewModel> certlist = cert;
foreach (var item in certlist)
{
string itemWithoutHour = item.CompletionDate.Value.ToShortDateString();
var itemConverted = DateTime.ParseExact(itemWithoutHour, "M/d/yyyy", null);
item.CompletionDate = itemConverted;
}
return certificateslist.GroupBy(x => x.ActivityTitle).Select(e => e.First()).ToList();
}

For any given DateTime object, you can reference its Date property to strip out the time values:
var withTime = DateTime.Now; // 8/7/2016 22:11:43
var withoutTime = withTime.Date; // 8/7/2016 00:00:00

The .NET framework does not have a date-only object.
It may be worth understanding how the DateTime structure works. Internally, it stores an offset in ticks (1 tick = 100 nanoseconds) since 1/01/0001 12:00 am in a single 64-bit unsigned integer. (1 tick = 100 nanoseconds)
The DateTime structure then provides many useful methods and properties for dealing with dates and times, such as adding some days to an existing date, or calculating the difference of two times. One useful property is Date, which rounds a DateTime object down to the nearest day (12:00 am).
Dates, times and dates-with-times are all very similar, the main difference is how you format them, a date-with-time where you omit the time is just a date.
What David has suggested is that you work with the DateTime structure internally, strip any times using the Date property, sort on the DateTime, compare them and modify them as DateTime objects.
Only convert them to a string when they need to be displayed, at which point you can use methods such as ToShortDateString() or ToString(string format) to display only the date.

Getting days between two dates with IQueryable

I have an IQueryable query which I need to use sometimes in the same method.
This query is based on another one which is passed as parameter.
I need to sum the result of a value multiplied by the number of days between two dates.
parameter query = IQueryable lista;
IQueryable<ChildEntity> query = lista.SelectMany(s => s.ChildEntities).Where(w=>w.IsActive.Equals("Y");
DateTime maxDate = lista.Max(m => m.Date);
decimal value = query.Sum(s => (s.Value) * (maxDate - s.ParentEntity.Date).Days);
which gives the exception:
Specified method is not supported.
I've also tried:
decimal value = query.Sum(s => (s.Value) * SqlMethods.DateDiffDay(maxDate, s.Parent.Date);
tried also SqlFunctions.DateDiff and EntityFunctions.DiffDays and all of these last three gives me this exception:
Method 'System.Nullable`1[System.Int32]
DateDiffDay(System.Nullable`1[System.DateTime], System.Nullable`1[System.DateTime])'
is not supported for execution as SQL.
I do not want to use Enumerable because this can result in a huge number of records.
Is there any other way to find a solution for this?
(by the way, I'm using Devart as provider.)

int days = 0;
DateTime today = DateTime.Now;
DateTime maxDate = lista.Max(m => m.Date);
While (today <= maxDate)
{
today.AddDays(7);
days++
}

Your problem is that the database you are running the query on doesn't have a time-span data type like net does, however most databases store a date as number of days since a official start date, this means that if you convert the dates to a number you can just do straight arithmetic on it
days = ((int)maxdate) - ((int)currentdate)
NOTE:thats sudo not runnable