Develop Reference

Converting a SAS Datetime value to datetime string in a Datatable using C#

I am working on an application that deals with loading some SAS datasets into SQL using C#. I use the SAS OLE DB Adapter to load the SAS dataset into C# data table and its working. However, the datevalues which is stored as string datatype in the SAS file is getting loaded as 10 digit number and stored as string format in the data table.
Hence, I tried to convert them to datetime however I am getting only the date value and time value is 00:00:00 for all the records. The source has the date and time values for each record. Below is my code that I use to convert the string to datetime string.
public string ComputeDate(string sSourceData)
{
string sProcessedDate = string.Empty;
int iDateValue = ((Convert.ToInt32(sSourceData) / 86400) + 21916);
Double dDate = Convert.ToInt32(iDateValue);
DateTime dtDateValue = DateTime.FromOADate(dDate);
sProcessedDate = dtDateValue.ToString();
return sProcessedDate;
}
The source data sample and the data that is getting loaded in the data table is below
The output after converting the OADate is below
It would be great if someone can help me out in providing me an solution for getting the output along with original time value provided in the source.
Thanks in advance.

Integer ratios such as
Convert.ToInt32(sSourceData) / 86400
will yield an integer.
Ensure the result contains the day fraction ([0..1]) which is the time part in SQL / C#
Double dDate = Convert.ToInt32(sSourceData) / 86400.0 + 21916.0;

Found the solution for my problem.
Instead of setting the datatype as Int, I need to set the datatype as double so that the timestamp of the OADate will be coming as decmimal values.
This way its getting output as datetime.
public string ComputeDate(string sSourceData)
{
string sProcessedDate = string.Empty;
Double iDateValue = ((Convert.ToDouble(sSourceData) / 86400) + 21916);
DateTime dtDateValue = DateTime.FromOADate(iDateValue);
sProcessedDate = dtDateValue.ToString();
return sProcessedDate;
}

Fastest way of retrieving records from Entity Framework DbSet on varchar Unix Timestamp

I have to retrieve all the records from a database that have been added since the last execution, this should happen daily.
The only thing that can identify those records from the rest is a Unix Timestamp (in milliseconds) or a Time (hhmmss) and a Date (yyyyMMdd). My problem is that all these columns are of type varchar.
The database is very big, and only getting bigger. Is there any way of getting only the rows with a Unix Timestamp higher than X without having to load the entire thing and parsing the timestamp?
What I do now is:
var sales = context.SALES.Select(s =>
new Sale {
Product = s.SC_PRDCT,
Terminal = s.SC_TERM,
Operator = s.MC_OP,
UnixString = s.SC_TIMESTAMP
})
.ToList()
.Where(m => terminals.ContainsKey(m.Terminal) && m.UnixTime > lastExecution);
public string UnixString
{
get { return unixString; }
set { unixString = value; UnixTime = long.Parse(value); }
}

Options that come to mind: If you have the ability to alter the schema while preserving the current fields I would consider adding a computed column to the database for a DateTime equivalent to the timestamp. Barring that, using a View to source the data for this type of search/report which can provide the translated timestamp.
If you don't have the ability to adjust the schema, then things will get a bit trickier. When you say the timestamp can be milliseconds or datetime in a string, does that mean the value can be something like either "1435234353353" (ms since Date X) or "20190827151530" for 2019-08-27 3:15:30 PM? If that is the case, as long as the length of the 2 strings, however formatted, is different then you can potentially still query against the field, it just won't be ideal:
Assuming the date option formatting is "YYYYMMDDHHMMSS":
string lastExecutionDate = targetDate.ToString("yyyyMMddHHmmss");
string lastExecutionMs = targetDate.ToUniversalTime().Subtract(
new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
).TotalMilliseconds.ToString();
var terminalKeys = terminals.Keys.ToList(); // You probably cannot use ContainsKey in Linq2EF...
var sales = context.SALES
.Where(s => terminalKeys.Contains(s.SC_TERM)
&& ((s.SC_TIMESTAMP.Length == 14 && s.SC_TIMESTAMP.CompareTo(lastExecutionDate) > 0)
|| (s.SC_TIMESTAMP.Length != 14 && s.SC_TIMESTAMP.CompareTo(lastExecutionMs) > 0 )))
.Select(s =>
new Sale
{
Product = s.SC_PRDCT,
Terminal = s.SC_TERM,
Operator = s.MC_OP,
UnixString = s.SC_TIMESTAMP
}).ToList();
If the SC_TIMESTAMP column only stores the timestamps in ms, and the time/date are in separate columns, then you don't need the conditional, just format your target datetime to a timestamp string (ms since 1970-01-01) and use that.
string lastExecutionMs = targetDate.ToUniversalTime().Subtract(
new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
).TotalMilliseconds.ToString();
var terminalKeys = terminals.Keys.ToList(); // You probably cannot use ContainsKey in Linq2EF...
var sales = context.SALES
.Where(s => terminalKeys.Contains(s.SC_TERM)
&& s.SC_TIMESTAMP.CompareTo(lastExecutionMs) > 0)
.Select(s =>
new Sale
{
Product = s.SC_PRDCT,
Terminal = s.SC_TERM,
Operator = s.MC_OP,
UnixString = s.SC_TIMESTAMP
}).ToList();
The caveat for this to work with the ms or datetime in the same field is that if you require timestamp or datetime is that the datetime string must be an ISO format "year Month Day Time24" which is sortable, otherwise it cannot be used in a comparison.

The lucky fact that the the unix timestamps are all equal in length, thus sortable, you can query them by the > operator in SQL.
Of course, as you' may have tried, m.SC_TIMESTAMP > lastExecution doesn't compile in C# but fortunately, EF6 (and EF3, more or less) translate the following predicate into the desired SQL predicate:
Where(m => m.SC_TIMESTAMP.CompareTo(lastExecution) > 0)
where lastExecution is a string.
Remember to add a comment to your code that this works until 2286-11-20 17:46:39, when the UNIX timestamp is 9999999999999. After that, your successors should use a more generic method that takes length into account.

EF DateTimes do not match saved values in SQL Server [duplicate]

This question already has answers here:
Milliseconds in my DateTime changes when stored in SQL Server
(4 answers)
Closed 7 years ago.
I save a datetime using C# Entity Framework, and when I load that time back from the database, the time varies from the value that I saved by 1 or more milliseconds.
Here is the C# Code:
public List<DateTime> TestDate()
{
var dates = new List<DateTime>();
DateTime testvalue = DateTime.Now;
dates.Add(testvalue);
IactexGMG2Entities firstContext = new IactexGMG2Entities();
var firstQuery = from p in firstContext.LocationProperties
where p.locationPropertyId == 4
select p;
var firstRec = firstQuery.Single();
firstRec.locationPropertyDateTime = testvalue;
firstContext.SaveChanges();
firstContext.Dispose();
IactexGMG2Entities secondContext = new IactexGMG2Entities();
var secondQuery = from p in secondContext.LocationProperties
where p.locationPropertyId == 4
select p;
var secondRec = secondQuery.Single();
var secondDate = secondRec.locationPropertyDateTime ?? DateTime.Now;
dates.Add(secondDate);
secondContext.Dispose();
return dates;
}
Here are the received values:
5/29/2015 5:43:25 PM . 154 , 635685182051540566
5/29/2015 5:43:25 PM . 153 , 635685182051530000
Here is the razor code that displays the values:
#foreach (var date in Model)
{
counter++;
<div>
#date . #date.Millisecond , #date.Ticks
</div>
}
As you can see, the second value, which was read back from the database, is lower than the first value by 1.0566 milliseconds.
The amount of variation varies, positive and negative, always with a small number of milliseconds.
Does anyone know how the conversion between the date values takes place?
Note: If I use the same context to read the date value, the values match. I assume that is because it is using the cached value, rather than the SQL Server value.

The problem is the different resolution between TSQL datetime and .NET DateTime data types
datetime only has a small resolution and is rounded to increments of .000, .003, or .007 seconds, whereas DateTime has a resultion of 100ns
Just use the new datetime2 SQL Data Type, which has the same resolution as .NET's DateTime, which is anyway recommended in new work, exactly for the issue you noticed

This actually has very little to nothing to do with Entity Framework. SQL Server as of 2008 has two DateTime types:
DateTime
Accuracy : Rounded to increments of .000, .003, or .007 seconds
DateTime2
Accuracy : 100 nanoseconds
Using Code-First Annotations you can set the property type like:
public MyClass
{
[Column(“CreatedOn", TypeName="DateTime2")]
public DateTime CreatedOn { get; set; }
}
Or using Fluent API:
modelBuilder.Entity<MyClass>()
.Property(p => p.CreatedOn)
.HasColumnType("DateTime2");

SQLite scalar function return type always string?

If I select a date-typed column in sqlite without any functions, it returns me a proper .NET DateTime object:
// Returns .NET DateTime object
select date_column from test_table;
But the moment I use a scalar function, it always returns me a string:
// Returns string
select datetime(date_column) from test_table;
select date(date_column) from test_table;
The reason I need this is for date rounding
// Rounds to month
select datetime(strftime('%Y-%M-01T00:00:00', date_column)) from test_table;
// Rounds to year
select datetime(strftime('%Y-01-01T00:00:00', date_column)) from test_table;
I tried implementing my own SQLiteFunction, but it still returns a string:
[SQLiteFunction(Name="round_date")]
public class RoundDate : SQLiteFunction
{
/// <summary>
/// Expects 2 arguments:
/// 1: date
/// 2: rounding (Month, Quarter, Year)
/// </summary>
public override object Invoke(object[] args)
{
DateTime date = SQLiteConvert.ToDateTime(args[0].ToString());
DateTime result = date;
switch (args[1].ToString())
{
case "Month":
result = new DateTime(date.Year, date.Month, 1);
break;
case "Quarter":
int quarter = (date.Month-1)/3;
var quarterStart = quarter*3 + 1;
result = new DateTime(date.Year, quarterStart, 1);
break;
case "Year":
result = new DateTime(date.Year, 1, 1);
break;
}
// function returns a DateTime object.
// But executing the statement returns a string
return result;
}
}
// Uses custom function, but still getting String
select round_date(date_column, 'Year') from test_table;
I know I can typecast/convert the string result manually.
But is there any way to force the select statement to return a DateTime object automatically?
[Edit 1]
I understand that Dates and DateTimes are not one of the natively-supported data types in sqlite.
I also know that I can manually convert the formatted strings to DateTime objects.
However, select date_column from test_table actually returns .NET DateTime objects. I assume the System.Data.SQLite implementation does this conversion automatically.
var selectStmt = sqlite.CreateCommand();
selectStmt.CommandText = "select date_column from test_table";
// ==============================
// ==============================
// Returns System.DateTime object
var this_is_a_DateTime_object = selectStmt.ExecuteScalar();
// ==============================
So I was expecting this same behaviour when using date(), datetime(), and my own SQLiteFunction implementation.

I don't believe there's a way to force the System.Data.Sqlite to automatically return the .NET DateTime datatype for you in this instance. Here's what's happening. Let's say your table definition looks like this:
CREATE TABLE test_table (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
date_column DATETIME NOT NULL
)
As you already noted, there's really no such thing as a data type in SQLite. But, as you also noted, the library is clearly handing you a native DateTime value back in this instance. My guess is the same as yours: it's using the semi-arbitrary wording of DATETIME as your request to undertake this helpful conversion on your behalf.
To test this, I added a clearly invalid date into one of my DATETIME columns and the library threw the exception "String was not recognized as a valid DateTime".
Now, to answer your question of why you don't get this same benefit using one of the date/time functions, the answer lies in the documentation:
Notice that all other date and time functions can be expressed in terms of strftime():
date(...) strftime('%Y-%m-%d', ...)
time(...) strftime('%H:%M:%S', ...)
datetime(...) strftime('%Y-%m-%d %H:%M:%S', ...)
julianday(...) strftime('%J', ...)
The only reasons for providing functions other than strftime() is for convenience and for efficiency.
So you're not really calling date/time functions. You're calling a formatting function whose return time is--you guessed it--a string. I made a few failed attempts myself at trying to cast the result of a function back to a datetime, like so:
select (date(...) as DATETIME), ...
But unfortunately, you can only typecast to one of SQLite's native storage classes using the type affinity name (one of: NONE, TEXT, REAL, INTEGER, or NUMERIC).
So it looks like you're back to having to take care of the conversion yourself after the fact. (As will I.)

As stated before, you can build a table with DATETIME type for a column, which returns a DateTime datatype in .NET.
Once you use any of the internal functions or Custom SQLiteFunctions it reverts it to an internal (non-date) data type.
There is a funny hack that can be used if you really want to avoid the String data type, this appears to be the ONLY way I can force it to return a DateTime data type in .NET after some level of date manipulation (so far). And I'm not really keen on it myself...
If you define a table like such...
CREATE Table test2(id bigint, name nvarchar(500), DT datetime)
The query result of the third column is a DateTime.
So following the below logic (again this is ugly, but maybe will inspire someone with more ideas) will return a date time that has been modified by the internal datetime function...
CREATE TABLE output (id bigint, name nvarchar, dt datetime)
;
INSERT INTO output
SELECT id, name, DateTime(dt,'+5 years') FROM Test2
;
SELECT * FROM output
;
DROP TABLE output
Executing this as a single statement will fill one datatable with the result as expected DateTime Datatype in column 3, shifted by 5 years.
In your code if you can use a unique output table name for each call then the approach is safe from conflict with multiple statements.

In SQLite databases there are no storage data type for "Date and time"
You can convert the datatype after a SELECT command to standard .Net DateTime format.
Query : "SELECT ID, DateModified FROM Customer"
string id = reader.GetInt64(0).ToString();
DateTime dateTime = (DateTime)reader[1];
SOURCE: http://techreadme.blogspot.de/2012/11/sqlite-read-write-datetime-values-using.html
I'm sorry the above information applies to SQLite versions 1 & 2. SQLite3 does include datatypes although they are implemented differently than Standard SQL Databases.
https://sqlite.org/datatype3.html

I don't use SQLite but, from what I've read, there is no native data type for dates and times. If your original query is producing a result set that the SQLite ADO.NET provider interprets as .NET DateTime values then I can only assume that it is based on format. In that case, you might be able to use strftime to output in that same format and get the data as DateTime values automatically. It appears that that format may be "yyyy-MM-dd HH:mm:ss.fff" in .NET terms.

Getting days between two dates with IQueryable

I have an IQueryable query which I need to use sometimes in the same method.
This query is based on another one which is passed as parameter.
I need to sum the result of a value multiplied by the number of days between two dates.
parameter query = IQueryable lista;
IQueryable<ChildEntity> query = lista.SelectMany(s => s.ChildEntities).Where(w=>w.IsActive.Equals("Y");
DateTime maxDate = lista.Max(m => m.Date);
decimal value = query.Sum(s => (s.Value) * (maxDate - s.ParentEntity.Date).Days);
which gives the exception:
Specified method is not supported.
I've also tried:
decimal value = query.Sum(s => (s.Value) * SqlMethods.DateDiffDay(maxDate, s.Parent.Date);
tried also SqlFunctions.DateDiff and EntityFunctions.DiffDays and all of these last three gives me this exception:
Method 'System.Nullable`1[System.Int32]
DateDiffDay(System.Nullable`1[System.DateTime], System.Nullable`1[System.DateTime])'
is not supported for execution as SQL.
I do not want to use Enumerable because this can result in a huge number of records.
Is there any other way to find a solution for this?
(by the way, I'm using Devart as provider.)

int days = 0;
DateTime today = DateTime.Now;
DateTime maxDate = lista.Max(m => m.Date);
While (today <= maxDate)
{
today.AddDays(7);
days++
}

Your problem is that the database you are running the query on doesn't have a time-span data type like net does, however most databases store a date as number of days since a official start date, this means that if you convert the dates to a number you can just do straight arithmetic on it
days = ((int)maxdate) - ((int)currentdate)
NOTE:thats sudo not runnable