i have a bit of a problem here, i am about to give it up myself, and i hope experts in here can help me salvage my data.
i have a program to collect some financial data. the format of data is the following
time, data
time, data
...
it is in a text format, i have about 30 files each around 1-2 GB. the problem i have is when i first start it, i have accidentally format it in a 12 hour time format "yyyy/MM/dd hh:mm:ss.fff" instead of 2400 hour time "yyyy/MM/dd HH:mm:ss.fff".
now my data is on a 12 hour format without am / pm.
i now need to write a program to convert the 12 hour format into 2400 format. the problems i face are the following:
the data is arranged in roughly a chronological order. the timestamp is from the data server and depends on the internet traffic, the order may be of 1 sec or so. but given the amount of data i have collected, it may be up to 100000 lines out of order.
without the am / pm, 1200 maybe 1200 am (which i need to subtract 12 hours from it) or 1200pm which i dont do anything.
the start of the file may start at any given time (i manually turn on the program to collect, so the time is not exact), so without looking ahead and see the day change, i would not know if 10:00 is am or pm
i try to convert the time before but unsuccessfully. can anyone help me by provide some code or pseudo code?
EDIT: Now i can word my problem exactly, i need to figure out if i am in AM mode or PM mode from the data, if there are not so many lines, i can probably use excel and correct it, manually look at the next few lines and deduce if it is AM mode or PM mode. With 1/2 billion lines, i need to programming it logically to determine if it is AM or PM mode, which i have difficulty doing so. and how do i deal with those few minutes that is out of order during transition time?
Assuming there are never more than 12 hours between records, your conversion program just needs to keep track of whether it's currently looking at AM or PM times. Then whenever the hour reaches 12 or wraps around to a lower number, you toggle the AM/PM flag. You'll have to provide a parameter telling it which mode to start in.
12am is 0000 and 12pm is 1200, but 1am is 0100 and 1pm is 1300 (So there is more to it than just subtracting 12 from 12am.
If you read in the time to a DateTime object, it is as simple as using the .AddHours(double)
myDateTime=myDateTime.AddHours(12);//Add 12 hours
//Or
myDateTime=myDateTime.AddHours(-12);//Subtract 12 hours
How you decide if you are in AM or PM mode is up to you, I don't quite follow why you cannot fix the problem since you said the data is generated at any time.
I would say if you run "yyyy/MM/dd hh:mm:ss.fff" (the 12hour format with no am/pm) on the string.
If it works you know that: The date is either still in am/pm mode OR in the AM in 24hour format
If it fails you know that: The date is in 24hour mode and it is PM (since it will be out of the range of the 12hour mode)
Using the format "yyyy/MM/dd HH:mm:ss.fff" should always parse your string and you know the following:
If it REALLY is an AM time, you have the correct date.
If the time is over 12:59, you will have the correct date for the PM(since it reported it in 24hour format)
If the date should be a PM time, but read in as an AM time, then you know you're in 12hour format and need to examine previous or following lines to try to determine if it should be AM or PM.
You can probably devise some sort of logic from that information and any other date/time information to get your desired results.
EDIT:
What you might be able to do is look at the file modified time (or the create time if you can) which should give you the AM or PM. You should then know if the first(or last) entry is AM/PM and can then proceed from there.
Example: If the time on the last entry was 11:00 (AM) and then next line is 3:00 for the same day, you would know that it is 3:00 (PM).
However, if the day changes you might not be able to tell if the entry starts on AM/PM. Looking at all the data for a given day might be able to give you the answer. If the day's entries are only from AM or only PM you will not be able to tell for certain.
Related
Let's say I have a date stored in UTC time in SQL using the datetime type. In C#, I am converting to the client using .ToLocalTime().
Will the time value itself be different depending on when I look at it? That is, during DST or not?
For example, the actual hour of this historic event (2022-09-30T16:00:00.00) is stored 4 hours ahead, so it converts as 12pm. This is correct. But when DST ends, and it's 5 hours ahead, will it show as being done at 11am? Is there a method to always show the actual hour it happened in the past, not adjusted?
No, because September 30 is always with DST. It doesn't matter if you check it in May or January, dates in September have always DST.
You can trivially confirm this yourself: Create a new DateTime object for 2022-02-03 (no DST) and print it today (October 4). Currently, we observe DST, but the date in February will still be output without DST, because DST is not applied in February.
Let's say I have the date 2014-11-2T00:00:00-0600 then I want to add 1 day, and have 2014-11-3T00:00:00-0700 .
Now if I do :
crtDay.Plus(Duration.FromStandardDays(1));
crtDay is a ZonedDateTime object
I end up with : 2014-11-2T23:00:00-0700 .
How do I move to the next day? (I could add hours until I moved to the next day, but it feels like there has to be a better way.)
Thanks.
(I'll assume from the values you gave that you are working in the US Mountain time zone.)
A "standard day" is 24 hours. However, on the day you mentioned, there are 25 hours in the calendar day, because the hour from 01:00 to 02:00 is repeated for the daylight saving time fall-back transition. Thus, you end up on the same day if you only add 24 hours.
In Noda Time, the ZonedDateTime structure is meant to represent a specific moment in time, with respect to a time zone. Thus, you can only move by linear instantaneous time, using Duration.
To move by calendar time, you'll have to take it's LocalDateTime and add a Period to it. If desired, you can then create a new ZonedDateTime from the result.
ZonedDateTime nextDay = zdt.LocalDateTime.PlusDays(1).InZoneLeniently(zdt.Zone);
Here I use InZoneLeniently for convenience. However, you should understand the behavior of that particular resolver. A different resolver (or a custom one) might be more appropriate, depending on your needs.
Say for example that you were starting at 1:30 on that day instead. The lenient resolver will assume you wanted to choose the latter of the two occurrences. Essentially you would be adding 25 hours even though 24 hours would still yield the same clock time. Perhaps your users might expect you to land on the first occurrence.
Also consider the spring-forward transition, where if you started at 2:30 then you'd land on date/time that doesn't exist. The lenient resolver will advance to the next valid clock time for the time zone - which might be acceptable, but it would have a different clock time of 2:00. Your users might expect you'd advance by a whole hour instead of just the next valid clock time.
The built-in resolvers are being reconsidered for Noda Time v2. See issue 295, and this alternate implementation.
Our webservice uses the Datetime.parse method to convert data from an xml to DateTime format. It parses Date and time strings separately and adds it together like this -
DateTime.Parse(Date_string).add(TimeSpan.Parse(Time_string)).
Code was working fine except for a few hours last week. Time was showing as 12 hours ahead of actual time. For example, 01/01/2011 10:00:00 will be parsed as 01/01/2011 22:00:00. Most of the requests during that time were processed with datetime values 12 hours ahead of actual time though some were processed correctly. It is working fine now and haven't seen it after that.
Has anyone seen a issue like this?
You say "Code was working fine except for a few hours last week", but you didn't specify exactly when that was or what time zone you are in. Any chance it was around a daylight savings time change?
You shouldn't use TimeSpan.Parse at all. A TimeSpan does NOT represent the time-of-day, despite its appearance as hh:mm:ss. A TimeSpan represents a fixed DURATION of time.
If you really are given separate date and time strings, join them together before parsing, such as:
DateTime dt = DateTime.Parse(date_string + " " + time_string);
You should also be aware of the timezone implications of the string you are sending in. See the MSDN article on DateTime.Parse for further details.
I have a question about DateTimeOffset and daylight savings time. To explain my question lets assume that right now the date and time is:
11/6/2010 10:15:00 AM
If I run this code:
DateTimeOffset myTime = DateTimeOffset.Now;
Console.WriteLine("Local time: " + myTime.ToLocalTime().DateTime);
Then I get this result:
Local time: 11/6/2010 10:15:00 AM
Meaning that the event happened at 10:15 in the morning (my time zone is Mountain Daylight Time (-6 offset)).
So, then I save this DateTimeOffset to my SQL Server 2008 db (as a DateTimeOffset). The next day I want to display it to the user. But now daylight savings has expired.
If I run the above WriteLine with the saved off value (from the previous day) what will display?
The offset stored in the database is -6. But now that daylight savings is over, the current offset is -7. So as I understand the documentation, it will first convert my time to UTC time (so it takes 10:15 AM and adds 6 hours (4:15 pm). It will then subtract the current offset of the local time (4:15 pm - 7 = 9:15 AM).
So if I my math is right, now when I display my event, it will show that it occurred at 9:15 AM rather than 10:15 AM.
This is not good. I want to store time zone information, but I need my times to stay static in the same time zone. (Meaning that if the event happened at 10:15 AM in Utah, then the next time I look at it (in Utah), I need to see that it was at 10:15 AM, regardless if the daylight savings time change has happened.
I can't think I am the first one to have this issue. What do others do to fix this? (Or do I have the facts wrong?)
No, it won't add the current offset - it'll add the offset at that date which is still -6. So it should still display 10:15AM, because it knows the date involved, and thus the time zone rules in force on that date.
You may well want to store a simple UTC time and the time zone identifier separately, by the way. If you're storing a time zone, then using DateTimeOffset won't be particularly helpful over just a UTC date/time. (On the other hand, it's clearer that it does represent an instant in time - DateTime is a horribly confused type which doesn't let you easily express what you're trying to represent.)
Of course I'd personally encourage you to look at Noda Time which in my very biased opinion is a rather clearer date/time API than the built-in one... but which isn't quite ready for production use. (We're getting there though...)
I have an application which records statistics. When a user submits their statistics the information is stored and the date/time is recorded as UTC e.g.
In the UK if I recorded figures at 03/08/2010 10:30 my recorded date/time would be 03/08/2010 09:30 as we are currently observing DST.
If I was in say Australia, and I recorded my statistics at the same time locally my recorded date/time would be 02/08/2010 23:30 as I think they are around 10 hours ahead.
My application needs to generate a statistical report at 7am every morning (local time) for everyone. So when it runs it does something like this:
var currentUtc = DateTime.UtcNow.AddDays(-1); // UTC date/time 24 hours from current
This date (time is ignored) is then used to pull the statistics from the database. Now this is working fine for the UK. It is other international countries I am finding the problems. More specifically Australia.
The issue I have is Australia are 10 hours ahead so my application is being triggered to run at around 03/08/2010 22:00 UK time to generate a report for 04/08/2010 07:00 Australia time. So what is happening is something like:
// the date/time at this point would be 03/08/2010 22:00 (UTC 03/08/2010 21:00)
var currentUtc = DateTime.UtcNow.AddDays(-1);
// so currentUtc is set to 02/08/2010 22:00 (which makes sense for UK, but not for Aussie)
If the above was a report for someone in the UK it means they would receive a report for 02/08/2010 on 03/08/2010 which is correct. However, for Australia (as they are 10 hours ahead) that means they would be receiving a report for 02/08/2010 on 04/08/2010 which would be 2 days behind.
What should be happening is when my application runs at 22:00 hours on 03/08/2010 it should be generating a report for Australia which is for 03/08/2010. However, as per the setup above you can see why this isn't happening.
I can't quite seem to put my finger on this one? I have a feeling it is something to do with my ignoring the time and just relying on the UTC date but I am not sure.
Because the user's day may not start and end at midnight UTC time, your date field in the database needs to include the time as well. Otherwise your Australian report will contain more than 24 hours of statistics. The date range from 02/08/2010 to 04/08/2010 only looks odd if you ignore the time. If you include the time and double-check the range, you'll see that it does contain 24 hours just like intended. When you convert it back to local time for displaying the report it will be fine.
the mismatch between "query at 7am local time" and "previous day UTC" seems to be odd IMHO - I'd imagine the user would instead want the previous day of data, with 'day' being in their local time, so the range you would want to search within is DateTime.Today.AddDays(-1) to DateTime.Today (so it covers 'yesterday, local time'), and since you're storing in UTC, we just add in ToUniversalTime to each of those to get them into UTC for querying the database.
var startOfToday = DateTime.Today.ToUniversalTime();
var startOfYesterday = startOfToday.AddDays(-1);
var query = ...