I am building a c# calendar application and have stored all the datetimes in Microsoft SQL-Server DateTime2 type. This data type is searchable using operators such as ">",">=" etc..
I have now read more on the subject for example these posts:
Daylight saving time and time zone best practices
How to store repeating dates keeping in mind Daylight Savings Time
Is it always a good idea to store time in UTC or is this the case where storing in local time is better?
I believe I have made an error when using UTC and dealing with different DST values and especially when related to future repeated events.
My current implementation works fine for events, until a repeating series goes over a DST time change.
I believe I will now need to store
local times,
the local timezone
and possibly UTC time
How should I structure my database and what data types should I use to store my data in the database that will support different client timezones and DST values whilst also allowing me to query for matches within specified start and stop datetime ranges?
For repeated events, you definitely need to store the time zone, yes, and I'd store the local date/time. You might also want to store the UTC value of the first occurrence, if that would be useful for comparison purposes. In theory you could just store the UTC occurrence of the first date/time, as that can be unambiguously converted to the local time (if you have the time zone) - but it's likely that you'll only ever need the local time, in which case performing those conversions may be pointless.
You should also consider how you want to handle changes in time zone data - because time zone rules do change, reasonably frequently. (It depends on the country, admittedly.) For example, for efficiency you may want to generate a certain number of occurrences and store the UTC date/time of each occurrence (having worked out what to do with skipped and ambiguous local times due to DST transitions) - but if the time zone data changes, you'll need to perform that generation step again for all repeated events.
Related
I'm hesitant to revisit this overly discussed topic, but I created a set of tables to store data and many of the tables included a field named "createdate" designated as "timestamp without time zone". The date/time values fed to these by the code base are always in UTC. It is intended that the UI will be able to control how the data is presented to the user, so some setting would dictate to convert to time zone in the UI.
Some of these timestamps will be for reports, so to display info for the end user. Other times, the values will be used to determine nightly jobs run against the data.
This is a typical is a multi-tenant cloud hosted system with clients across different time zones.
The server should never be moved, although I suppose changing hosting zones is a very remote possibility.
It was written on the .net platform. Not using noda time, just the built in DateTime stuff (for now).
The docs are pretty clear how the timestamp with time zone stores the info:
https://www.postgresql.org/docs/current/datatype-datetime.html
This answer also has good background on the difference in the two main timestamp data types:
https://stackoverflow.com/a/14616640/1905693
This answer also has some good info, but is Java-oriented:
What is the most recommended way to store time in PostgreSQL using Java?
And Josh Berkus has a dated article that was helpful:
https://it.toolbox.com/blogs/josh-berkus/zone-of-misunderstanding-092811
It seems most of these recommend the timestamp with time zone, but in my case, is timestamp without time zone appropriate?
If I did want to rely on pg to do the conversion, would the AT TIME ZONE clause be ok?
From an overall system architecture, is relying on the UI to change presentation a common and reasonable approach? (yeah, this one may be too opinion-flavored for SO's format)
Not a moment
As others said, TIMESTAMP WITHOUT TIME ZONE is the wrong data type.
That type holds only a date with time-of-day, for example noon on January 21st of 2021. But we cannot know if that means noon in Tokyo Japan, noon in Toulouse France, or noon in Toledo Ohio US — three very different moments, several hours apart from one another. So this type cannot represent a moment, is not a specific moment on the timeline.
The TIMESTAMP WITHOUT TIME ZONE type is good for three kinds of use-cases:
Representing multiple moments all known the same in their locality. For example, Acme Corp orders the manager at each factory in Delhi, Düsseldorf, and Detroit, to make an announcement in two days at their local time of noon.
Representing a date and time-of-day where the intended time zone is unknown. I consider this faulty data that should be rejected. But if you insist on writing it to the database, this type would be appropriate.
Booking future appointments where we want to keep the time-of-day even if those pesky politicians change the offset of the time zone(s) in their jurisdiction. These political changes happen surprisingly often. So book an appointment using two columns: TIMESTAMP WITHOUT TIME ZONE in one, and the name of the intended time zone in another. Time zones are named with Continent/Region format such as Africa/Tunis. At runtime, when you need a moment for calendaring, apply the time zone to the date and time to dynamically determine a moment according to the now-current time zone rules. In Noda Time, you would retrieve a LocalDateTime and time zone, to produce a ZonedDateTime for calendaring.
Moments
When you care about moments, specific points on the timeline, use TIMESTAMP WITH TIME ZONE.
In Postgres and many other databases, this type has a bit of a misnomer. The time zone is not actually saved with the date and time. Instead, upon submission to the database, any indicator of time zone or offset-from-UTC is used to adjust to UTC (an offset of zero hours-minutes-seconds). That UTC value is what Postgres writes to the database. And UTC is what Postgres always retrieves from the database.
Beware: Some middleware and tooling has the well-intentioned but very confusing anti-feature of dynamically applying a default time zone to the retrieved value, adjusting from UTC to that time zone. This creates the illusion of that time zone having been saved with the data. But, no, not so. A TIMESTAMP WITH TIME ZONE column stores only UTC values.
Use case examples:
Tracking the moment when a database record was created, modified, or deleted.
Logging for debugging or sysadmin work.
Tracking when a critical contract is signed or comes into effect.
Like you have seen repeatedly in the sources you quote, timestamp with time zone (timestamptz) is the more appropriate choice in all cases where different time zones may be involved.
Internally, storage is the same. But with timestamptz, Postgres already knows where timestamp value applies, while it is just oblivious in this regard with timestamp without time zone (timestamp).
Note that the given time zone is not stored. That's a common misunderstanding. It's only used to compute UTC time. See:
Time zone storage in data type "timestamp with time zone"
AT TIME ZONE is the SQL construct to use to get the corresponding time for a given time zone.
SELECT my_timestamptz AT TIME ZONE 'US/Hawaii';
With timestamp it gets awkward:
SELECT my_timestamp AT TIME ZONE 'UTC' AT TIME ZONE 'US/Hawaii';
Also, when using timestamp any appended time zone information in literals is ignored, which is typically not what you want.
See:
Ignoring time zones altogether in Rails and PostgreSQL
I usually set the timezone of the role I am using?
ALTER ROLE my_db_user IN DATABASE my_database
SET "TimeZone" TO 'UTC';
Seems to help getting the DateTime correctly.
We have an application related to aviation, and specifically flights.
Times have to be stored in local, so I chose to use UTC Time + an offset, but then now I realize it is a bad choice:
By storing the timezone as an offset, I lose track of the original timezone and this has implications when dealing with daylight savings.
For example I can store a time in Alpine, UT as UTC time and a -6 offset and also a time in Phoenix, AZ as UTC time and a -6 offset.
But when daylight saving comes, the time will change in Alpine, but not in Phoenix.
So, I need to store the proper timezone and I have seen that there are also different lists with a different syntax, so I am assuming there are different standards.
In C#, what would be the best option to store a local time with the local time zone to make it work with daylight saving changes?
From the discussion in the question's comments, I understand that you are working with flight time schedules - that is, the time a future flight is intended to depart. This is indeed a case where the local time is more important than the UTC time.
Since you have the local time and location of departure (ex: 5:00 PM in Salt Lake City), then you should be storing in your database of scheduled departure times two values:
17:00 - The relevant local time of the departure
SLC - The location where the time is relevant
If this is a specific occurrence of this flight, then you should store the date as well:
2018-06-01T17:00 - The specific relevant local time of the departure
SLC - The location where the local time is relevent
These are the details that are contextually relevant to your business use case. Do not lose sight of them by converting them to UTC.
That said, you might consider storing them as a DateTimeOffset (2018-06-01T17:00-06:00), which makes converting to UTC trivial for a given instance. However there are two problems with this approach:
It cannot work with recurrences, because the offset may change.
Even for a single instance, the offset might change - if the government controlling the time zone decides to change their standard offset or daylight saving time rules before that occurrence takes effect. If you do take a DateTimeOffset approach, or a UTC-based approach, you must be prepared to recalculate future events in the face of such changes. (For more on this, see my blog articles: On the Timing of Time Zone Changes and Time Zone Chaos Inevitable in Egypt. There are countless other examples.)
With regards to the location - because you are working with data that is contextually applicable to the airline industry, I recommend using IATA airport codes like the SLC that I showed above. In other contexts, one might store an IANA time zones identifier like America/Denver, or a Windows time zone identifier like Mountain Standard Time.
You may find my "Airport Time Zones" gist (code and output table) useful for working with IATA airport codes. You'll have to decide how that data will flow through your system. If you are running on Windows and want to use the TimeZoneInfo class to convert times to different time zones, then use the Windows time zone IDs shown there. If you want to use the IANA time zone IDs instead, consider using Noda Time, or you can use my TimeZoneConverter library. There are several different options here, so explore them all carefully and pick ones that make sense to you.
Noda Time would be a great choice, IMHO. Not only would you get excellent time zone support, but you'd also be able to use types like LocalTime or LocalDateTime which align well with the scenarios described.
As I wrote in my comment, do not store local dates. Instead, store datetime values as UTC, and convert to local datetime when you need to display it.
You can use the ConvertTimeFromUtc method of the TimeZoneInfo class for that.
This means you will have to also keep a list of locations and whatever TimeZoneInfo they are associated in - For example,
Jerusalem would be associated with Israel Standard Time,
Rome with W. Europe Standard Time,
Hawaii with Hawaiian Standard Time
and so on. (I'll bet you can find such a list online somewhere.)
Please note that the ConvertTimeFromUtc method handles the daylight savings problem for you as well.
Then you can do something like this to get the local time by location:
DateTime GetLocalDateByCityName(DateTime utc, string cityName)
{
var timeZoneInfoId = GetTimeZoneInfoIdByCityName(string cityName);
return TimeZoneInfo.ConvertTimeFromUtc(utc, TimeZoneInfo.FindSystemTimeZoneById(timeZoneInfoId);
}
And of course, in GetTimeZoneInfoIdByCityName you get the TimeZoneInfoId for the specific city.
You should use TimeZoneInfo.ConvertTime to convert DateTime between time zones if you need to handle Daylight Savings.
TimeZoneInfo.ConvertTime(myDateTime, timeZone);
TimeZoneInfo.ConvertTime(myDateTime, fromTimeZone, toTimeZone);
The back-end is C# and SQL Server.
I save logging activity date times from different time zones in UTC in my DB. I need to populate a globally distributed front end so it localizes the logged UTC time based on where the front end is located, which will be known e.g. PT, ET. It also needs to handle daylight savings time which is where I'm running into difficulty.
As the UTC offset changes based on the time of the year because of daylight savings time, is there a way for me to find out what the UTC offset is in C# given that I know the date/time and the time zone.
is there a way for me to find out what the UTC offset is in C# given that I know the date/time and the time zone
Technically you shouldn't need to find out this information, the underlying system clock/calendar/what-have-you should take care of it. (I'm sure anybody who has worked on date-time logic will attest to the fact that it is non-trivial to say the least.)
Ideally, any time you need to convert a time zone, you have three things:
A DateTime object representing the value to be converted.
A TimeZoneInfo object representing the known current format of the DateTime object (in this case UTC).
A TimeZoneInfo object representing the target time zone.
Given those things, any time you want to display the DateTime to a localized user, you'd use something like TimeZoneInfo.ConvertTime to perform the conversion. This should ideally be done as close to the UI as possible so that backing logic is always consistently performed in UTC.
I'm reading a lot about timezone, offset, utc, local time, javascript functions, DST, bacon and I 'm trying to put this all together to build a solid/correct structure for my app.
Suppose my app is something like StackOverflow.
That is how I'm doing ...
The server is in another country, so I set it to UTC 00:00.
I'm storing date as DateTimeOffset.
I'm not storing TimeZoneID.
Date is being sent to the client in this format: 2012-07-19T14:30:00-03:00.
I'm using angular filter to convert it to local time.
I have a few questions about it ...
1º Server TimeZone?
About my server (single server) ... should it be running with a "neutral" UTC (+00:00)? And what if, in the future, we move to a farm where servers run on different locations?
2º What should I store?
Currently, I'm storing just date as DateTimeOffset. I'm reading about saving the TimeZoneID but I'm seeing no use at all for this. Am I missing something?
Or should I store date as DateTimeUtc with a TimeZoneID and manually convert every date with the TimeZone class?
3º How to convert to local time?
Is it safe to convert data on the client? Or date conversions should be always on the server side?
4º About DST.
Using my current approach. Will DST be respected?
One very important thing to understand about date/time is that there is no one right way for doing everything. The common answer "use UTC" is not always applicable. Context is very important, and there are different techniques and approaches based on what the values you are working with are representing. If you could elaborate on what they are used for in your application, I will update my answer accordingly. In the meantime, I'll try to address the specific points you have brought up already:
#1 - Server Time Zone
Keeping your server at UTC is a best practice, and it is what you can expect from cloud providers like Azure or AWS also. But it isn't something that you should be dependent on. Your server should be able to be set to any time zone without it affecting your application. As long as the clock is in sync with an NTP server, choice of time zone should not matter.
So how do you ensure that? Simple, just make sure your application avoids all of the following:
DateTime.Now
DateTimeKind.Local
TimeZone (the entire class)
TimeZoneInfo.Local
DateTime.ToLocalTime()
DateTime.ToUniversalTime() (because it assumes the input is local)
Misc. other methods that assume a local input or output, such as TimeZoneInfo.ConvertTimeToUtc(DateTime) (this particular overload doesn't take a time zone, so it assumes the local time zone)
See also my blog post: The Case Against DateTime.Now.
Note that I didn't include DateTimeOffset.Now in the list. Although it's a little bit of a design smell, it is still "safe" to use.
#2 - What to store
I suggest you read my answer to DateTime vs DateTimeOffset. It should clarify some things. Without regurgitating the whole thing, the main point is that while both represent a point in time accurately, a DateTimeOffset provides perspective, while a UTC DateTime does not.
You also asked when you should store a TimeZoneInfo.Id. There are at least two scenarios where this is required:
If you are recording events in the past or present, and you plan on allowing modifications to the recorded timestamps. You need the time zone to determine what the new offset should be, or how the new input converts back to UTC.
If you are scheduling time out into the future, you will need the time zone as part of the recurrence pattern (even for a single occurrence). See here and here also, (while for other languages, the same principles apply).
Again, the exact answer depends on what exactly the timestamps represent. There is no one ring to rule them all.
#3 - Client Safety
If it's a .NET client, sure you can convert there. But I think you are asking about a JavaScript client browser.
"Safe" is a relative term. If you're asking for exact perfectness, then no. JavaScript isn't safe for that, due to an error in the ECMAScript specification (ES1 through ES5.1. It is being worked on for ES6). You can read more in my blog post: JavaScript Date type is horribly broken.
However, if you are working with relatively current data, and the users of your application are not in a part of the world where time zones are volatile, or you don't require precise results 100% of the time, then you can "safely" use JavaScript to convert to the user's local time zone.
You might avoid some of these issues with libraries that implement the IANA TZDB in JavaScript, such as the ones I list here. But many of them are still dependent on JS Date, so they still have issues. (Side note - I'm working on a JS library that will counter this, but it is not ready to share yet).
Conversions on the server side are a much better choice, as long as you can ask the user for their time zone. Most of the time, I think this is doable.
You might consider asking using a map-based timezone picker, such as this one or this one. Both of which will require you use IANA time zones, which for .NET means using Noda Time, which is a great idea anyway (IMHO).
#4 - Daylight Saving Time
With your current approach, DST will be respected within the definition of the current DST rules for the time zone the user has set for their local browser. (Again, refer to my blog post for why this is the case).
A conversion from any value with an offset (whether -03:00 or Z) that passes through the Date object (which I believe an Angular filter will do), will properly convert to the specific unix timestamp.
The errors that would crop up with DST conversions for prior DST rules are because going from the unix timestamp inside the Date object to the local time zone will always assume that the current DST rule is applicable, even if the time fell into a period that had a different rule.
This actually depends on the actual application you're writing, but the most simple/robust approach IMO is to store/compute all your dates using UTC, and convert it back to the local time zone when you display it to the user.
I'm currently working on a project that requires a user (in their respective time zone) to set a 'cut off' time for certain Notifications that they receive throughout the day. The user will, for instance, say they want no alerts prior to 8:00 AM and no later than 7:30 PM. This user is located in CST (the time zone information for each user is saved) and the 'server' is in EST. The server is where all of the logical decisions occur for which user should be notified, etc.
What would be the best practice to convert the time part only for comparison purposes?
My current thought would be to take the time set by the user, say "7:00 AM", and create a full DateTime object and store that as .ToUniversalTime(). On the server I can convert the DateTime object to Local time and compare the Time portion only.
Does anyone have a 'better' solution or possibly any advice from someone who has tackled a similar issue?
Thanks!
I suggest that you store the minimum time and maximum time as local times, without dates. You will also want to store the users general timezone information, for example "Central Time", not CST or CDT.
When you are ready to make a determination about sending a notification, convert the current UTC time to the user's local timezone and compare the stored time range against the converted time.
Trying to create a full calendar date from stored time information will be problematic in certain cases, specifically for daylight savings changes. By converting from UTC to user's local time for the comparison, that problem will be avoided.