zic bug: Wall-clock ordering of transition times
I noticed a bug while incorporating changes to commentary for the Turks and Caicos Islands… https://mm.icann.org/pipermail/tz/2020-December/029609.html Our contemporaneous use of "3:00" in the UNTIL field for the 2018-03-11 change intentionally avoided two clock changes in a row (first from 01:59:59 AST to 01:00:00 EST, then from 01:59:59 EST to 03:00:00 EDT an hour later). https://github.com/eggert/tz/commit/8792d7bed2c684cf842592fd7779645f335a33f6 My recently-added commentary clarifies the reasoning behind this and concludes that the order was likely intended to take effect from 03:00 -04, once the Eastern US had sprung forward, and not at 02:00 -04 as the text suggests. However, I noticed that using "2:00" in the UNTIL field instead doesn't actually result in the expected extra transition: --- to2050.tzs 2020-12-05 14:11:12.000000000 -0500 +++ to2050new.tzs 2020-12-05 14:17:31.000000000 -0500 @@ -5201,7 +5201,7 @@ 2014-03-09 03 -04 EDT 1 2014-11-02 01 -05 EST 2015-03-08 03 -04 AST -2018-03-11 03 -04 EDT 1 +2018-03-11 02 -04 EDT 1 2018-11-04 01 -05 EST 2019-03-10 03 -04 EDT 1 2019-11-03 01 -05 EST It seems that, the 02:00 wall-clock time having already been approached once for the transition out from AST, zic assumes that the transition specified in the US Rule at the same wall-clock time has also taken simultaneous effect in Eastern Time, although in reality it wouldn't yet for another hour. In this case, the wall-clock times of these two transitions are identical, while in actuality the transitions are meant to take effect sequentially. It appears that this bug also manifests in other cases where the strict ordering of transitions does not exactly match the strict ordering of wall-clock times: Trying out "2:30" in the UNTIL field should mean that the Zone transition should happen 30 minutes BEFORE the US Rule would take effect (and thus require two transitions), but by wall-clock time, it would "appear" to be 30 minutes AFTER (fooling zic into thinking only the "latter" Zone transition is needed as the Rule would have "already" taken effect). And, indeed, the results are similarly incorrect here, too: --- to2050.tzs 2020-12-05 14:11:12.000000000 -0500 +++ to2050new.tzs 2020-12-05 14:36:07.000000000 -0500 @@ -5201,7 +5201,7 @@ 2014-03-09 03 -04 EDT 1 2014-11-02 01 -05 EST 2015-03-08 03 -04 AST -2018-03-11 03 -04 EDT 1 +2018-03-11 02:30 -04 EDT 1 2018-11-04 01 -05 EST 2019-03-10 03 -04 EDT 1 2019-11-03 01 -05 EST Certainly an edge case in this particular instance, but given that many historical transitions coincide tightly with transitions in other regions being "joined" or "left", perhaps one worth a further look. -- Tim Parenti
On 2020-12-05 19:45, Tim Parenti wrote:
It appears that this bug also manifests in other cases where the strict ordering of transitions does not exactly match the strict ordering of wall-clock times: Trying out "2:30" in the UNTIL field should mean that the Zone transition should happen 30 minutes BEFORE the US Rule would take effect (and thus require two transitions), but by wall-clock time, it would "appear" to be 30 minutes AFTER (fooling zic into thinking only the "latter" Zone transition is needed as the Rule would have "already" taken effect). And, indeed, the results are similarly incorrect here, too:
--- to2050.tzs 2020-12-05 14:11:12.000000000 -0500 +++ to2050new.tzs 2020-12-05 14:36:07.000000000 -0500 @@ -5201,7 +5201,7 @@ 2014-03-09 03 -04 EDT 1 2014-11-02 01 -05 EST 2015-03-08 03 -04 AST -2018-03-11 03 -04 EDT 1 +2018-03-11 02:30 -04 EDT 1 2018-11-04 01 -05 EST 2019-03-10 03 -04 EDT 1 2019-11-03 01 -05 EST
Certainly an edge case in this particular instance, but given that many historical transitions coincide tightly with transitions in other regions being "joined" or "left", perhaps one worth a further look.
Wow! I do not think that this is a bug. It could be the rule that governs the cases where the change specified by two successive Zone lines Z₁ followed by Z₂ does not cause a switch to the time scale as specified by the STDOFF of Z₂ plus the SAVE value as specified with the RULE of Z₂ (as one would expect). That such cases exist is well-known; one example is the jump in America/Juneau near 1980-04-27 that uses YDT = UTC - 08 h starting at 1980-04-27T10Z, one hour before YDT starts to be used according to the US Rules. I do not have a complete list of these cases, nor do I know whether the following precisely formulated rule, as suggested by the examples you describe, would explain all these cases. Let X be the instant designated by the UNTIL field of Z₁; namely, the first instant at which the time scale UTC or UTC + F of UTC + F + S, (depending on the explicit of implicit suffix u, s, or w in the UNTIL field of Z₁) assumes a value >= the datetime indicated in the UNTIL field of Z₁, where F is the value of the STDOFF of Z₁ and S is the SAVE value last set strictly before the instant X. Then such a case would occur if the following two conditions apply: • the RULE field of Z₂ names a Rule that does not contain a Rule line applicable (in the context of the STDOFF of Z₂) at the instant X but that does contain a Rule line applicable at an instant Y next after X, and that Rule line specifies an increase in the SAVE value by an amount B > 0 h; • the instant Y is at most B after X (that is, UTC(Y) - UTC(X) <= B) and Y is strictly before the instant designated by the UNTIL field of Z₂. Under these conditions, the SAVE value used for the instants on or after X, and strictly before Y is the one that is set only later at the instant Y (and which is greater by B than the SAVE value described by the Rule of Z₂ for the instants in [X..Y[). Probably, also the LETTER value and the dst bit as set at Y also apply to the instants in [X..Y[. I find it a bit embarrassing that the precise semantics of zic input is only known to C compilers, but I have to admit that I am currently not seduced by the prospect of wading through C code maintained since 1980 or so, with lots of macros. Regarding the wording of the ordinance: • it is not unambiguously evident which time scale is used for interpreting the expression "come into operation at 2:00 a.m. on 11th March 2018"; it could be the time scale used earlier (as often used in tzdb) or the one used afterward; • that the time scale "as the Eastern United States of America" does not assume the value 2018-03-11T02:00 is an interpretation that assumes that this time scale is right continuous (as a function of UTC). That civil time scales are right continuous is an assumption that is supported by the general method of adjustment with the help of time signals (and valid for UTC as a function of TAI) but it is nowhere stated explicitly in any legal text, as far as I know. The text of the ordinance is consistent if one assumes that the time of the switch is given in EST and EST/EDT as well as the time scale of Grand Turk is assumed to be left continuous when UTC is 2018-03-11T07Z. Michael Deckers.
On 12/6/20 10:06 AM, Michael H Deckers via tz wrote:
I find it a bit embarrassing that the precise semantics of zic input is only known to C compilers
We should document zic's behavior better, yes. As I vaguely recall, it's done this way on purpose and so is a feature and not a bug. Perhaps Arthur's less-fallible memory can cite chapter and verse about when and why the behavior was put into zic. I gave the documentation a shot by installing the attached patch into the development version. Further improvements would be welcome.
We should document zic's behavior better, yes. As I vaguely recall, it's done this way on purpose and so is a feature and not a bug. Perhaps Arthur's less-fallible memory can cite chapter and verse about when and why the behavior was put into zic.
There is indeed some of this that's done on purpose; it's in the function "writezone" in the "zic.c" file, in particular in the section headed by the comment "Optimize." That section has grown more elaborate over the years. Elaboration began in May of 1996 (at which time the elaboration included a "Horrid special case" comment). The web page... https://mm.icann.org/pipermail/tz/1996-May/date.html ...includes links to messages regarding the work done in 1996; check out the messages including "simultaneous" in the subject. --ado On Mon, Dec 7, 2020 at 2:21 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
On 12/6/20 10:06 AM, Michael H Deckers via tz wrote:
I find it a bit embarrassing that the precise semantics of zic input is only known to C compilers
We should document zic's behavior better, yes. As I vaguely recall, it's done this way on purpose and so is a feature and not a bug. Perhaps Arthur's less-fallible memory can cite chapter and verse about when and why the behavior was put into zic.
I gave the documentation a shot by installing the attached patch into the development version. Further improvements would be welcome.
On 12/7/20 1:29 PM, Arthur David Olson wrote:
https://mm.icann.org/pipermail/tz/1996-May/date.html <https://mm.icann.org/pipermail/tz/1996-May/date.html> ...includes links to messages regarding the work done in 1996; check out the messages including "simultaneous" in the subject.
Thanks for the pointers. So it's a bit of code that you and I wrote, and that I'd totally forgotten about. Good memory! For the record, the commit in question is dated 1996-05-16 13:48:28 -04 and can be found here: https://github.com/eggert/tz/commit/2f53da99b5fb6e0a4cf5861ed80d84643032aeec and it says (in C) roughly what I attempted to write (in longer English) in the recent doc patch. Admittedly a bit of a delay between installing the code and installing the corresponding documentation....
Good memory!
I wish. In this case, though, grep and a comprehensive text file of time zone electronic mail sufficed.-) --ado On Mon, Dec 7, 2020 at 8:13 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
On 12/7/20 1:29 PM, Arthur David Olson wrote:
https://mm.icann.org/pipermail/tz/1996-May/date.html <https://mm.icann.org/pipermail/tz/1996-May/date.html> ...includes links to messages regarding the work done in 1996; check out the messages including "simultaneous" in the subject.
Thanks for the pointers. So it's a bit of code that you and I wrote, and that I'd totally forgotten about. Good memory!
For the record, the commit in question is dated 1996-05-16 13:48:28 -04 and can be found here:
https://github.com/eggert/tz/commit/2f53da99b5fb6e0a4cf5861ed80d84643032aeec
and it says (in C) roughly what I attempted to write (in longer English) in the recent doc patch. Admittedly a bit of a delay between installing the code and installing the corresponding documentation....
(+howard.hinnant@gmail.com, for reference to his https://github.com/HowardHinnant/date library below) Hello folks, I maintain my own TZ database parser and compiler for one of my projects. I run automatic validation tests against Howard's date library, so I was curious about how my TZ parser and Howard's parser handle this 2:00 versus the 3:00 UNTIL time for America/Grand_Turk. The summary is that both of them handle the 2:00 UNTIL time just fine. I did not write any special case for Grand_Turk in my parser, so I was curious to know why it just worked, just in case it worked only "accidentally". It took me some time to understand my own code, since I cannot keep all of its parsing logic in my head for more than a few weeks at a time. The core of why it "just works" seems be in the interpretation of the "2:00 transition to US Rule", and the fact that "2:00" is Wall-time, not S-time or U-time: So, on Mar 11, 2018: * At 1:59, Grand_Turk is on UTC-4. It does not know about Eastern Time, nor does it care. * At 2:00, Grand_Turk switches to UTC-5, and starts using US Rules. One of those US rules says, at 2:00 *wall time*, use SAVE +1. * Grand_Turk says, ok, I used to be on UTC-4. Now the rules say that I should be UTC-5+1 (i.e. UTC-4), so I'll just keep using UTC-4. The wall clock goes from 1:59, to 2:00, then 2:01, ..., 2:59, then finally to 3:00. * Grand_Turk only cares about transition rules and UTC offsets. It does not know about Eastern Time, so it does not change 2:00 to 1:00. * The time interval 2:00-3:00 does not exist for clocks on Eastern Time. But Grand_Turk is not using an Eastern clock, it's using its own clock, and 2:00-3:00 *does* exist for Grand_Turk. Anyway, that's my simple human translation of what my code does for this case. The actual logic, just like zic.c, is far more complicated than can be explained in words. I am not familiar with the details of what Howard's date library does internally, but I'm always happy to be in agreement with his library. Regards, Brian On Mon, Dec 7, 2020 at 5:25 PM Arthur David Olson < arthurdavidolson@gmail.com> wrote:
Good memory!
I wish. In this case, though, grep and a comprehensive text file of time zone electronic mail sufficed.-)
--ado
On Mon, Dec 7, 2020 at 8:13 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
On 12/7/20 1:29 PM, Arthur David Olson wrote:
https://mm.icann.org/pipermail/tz/1996-May/date.html <https://mm.icann.org/pipermail/tz/1996-May/date.html> ...includes links to messages regarding the work done in 1996; check out the messages including "simultaneous" in the subject.
Thanks for the pointers. So it's a bit of code that you and I wrote, and that I'd totally forgotten about. Good memory!
For the record, the commit in question is dated 1996-05-16 13:48:28 -04 and can be found here:
https://github.com/eggert/tz/commit/2f53da99b5fb6e0a4cf5861ed80d84643032aeec
and it says (in C) roughly what I attempted to write (in longer English) in the recent doc patch. Admittedly a bit of a delay between installing the code and installing the corresponding documentation....
On 2020-12-29 19:08, Brian Park wrote:
The core of why it "just works" seems be in the interpretation of the "2:00 transition to US Rule", and the fact that "2:00" is Wall-time, not S-time or U-time:
Then why does your code work for the transition at 1991 Mar 31 2:00s in Kaliningrad? Michael Deckers.
Is there a known special problem for Europe/Kaliningrad? My code and the HowardHinnant date library agree on this transition. The TZ database says: Zone Europe/Kaliningrad 3:00 Russia MSK/MSD 1989 Mar 26 2:00s 2:00 Russia EE%sT 2011 Mar 27 2:00s Rule Russia 1984 1995 - Sep lastSun 2:00s 0 - Rule Russia 1985 2010 - Mar lastSun 2:00s 1:00 S So, on Mar 26, 1989: * at 1:59, Kaliningrad is on UTC+3 (SAVE=0) * at 2:00s (which is identical to 'w' in this case, because we need to use the STDOFF of the transition immediately to the left), Kaliningrad switches to UTC+2+{SAVE} * but the Russia rule also takes effect at 2:00s, so SAVE becomes +1 * so Kaliningrad switches to UTC+2+1, i.e. remains at UTC+3 * Kaliningrad clock stays at 2:00 I think the key here is that both the Zone entry and the Rule entry use the same "2:00s", which are both interpreted using the STDOFF of the transition immediately to the left of the 2:00s. I don't know the internal details of the HowardHinnant's date library, but my code normalizes the various transition times to 'w' time, so the above explanation is only a rough translation of what the code actually does. The other interesting question is what happens on Sept 24, 1989, when it reverts back to STD time at 2:00s. Both my code and HowardHinnant date library seem to correctly convert the 2:00s to 3:00w, and the clock goes from 2:59 -> 2:00. Brian On Tue, Dec 29, 2020 at 11:58 AM Michael H Deckers < michael.h.deckers@googlemail.com> wrote:
On 2020-12-29 19:55, Michael H Deckers wrote:
1991 Mar 31 2:00s
instead of 1989 Mar 26 2:00s
Sorry.
Michael Deckers.
On 2020-12-29 21:32, Brian Park wrote:
Is there a known special problem for Europe/Kaliningrad?
No. In my mail, I tried to say that your assertion >> the fact that "2:00" is Wall-time, not S-time or U-time: is irrelevant for the issue because Kaliningrad has an "S-time" when two transitions are coalesced. Michael Deckers.
Then I wish you had said exactly that, instead of having me go off on a tangent. Maybe I am misinterpreting something, but I don't understand why you are being confrontational about this. I am presenting results from 2 different TZ parsers, using algorithms developed independently. I thought this would be interesting and relevant to this topic. I don't understand the point that you are trying to make. On Tue, Dec 29, 2020 at 2:33 PM Michael H Deckers < michael.h.deckers@googlemail.com> wrote:
On 2020-12-29 21:32, Brian Park wrote:
Is there a known special problem for Europe/Kaliningrad?
No. In my mail, I tried to say that your assertion
>> the fact that "2:00" is Wall-time, not S-time or U-time:
is irrelevant for the issue because Kaliningrad has an "S-time" when two transitions are coalesced.
Michael Deckers.
On 2020-12-29 22:45, Brian Park wrote:
Then I wish you had said exactly that, instead of having me go off on a tangent.
So do I. I am sorry to have lead you astray.
Maybe I am misinterpreting something, but I don't understand why you are being confrontational about this. I am presenting results from 2 different TZ parsers, using algorithms developed independently. I thought this would be interesting and relevant to this topic.
I do not want to sound confrontational but I do want to be short. My comment on your original post concerned only one sentence, and I did in no way comment on the rest of that post.
I don't understand the point that you are trying to make.
The complete sentence I commented about is: The core of why it "just works" seems be in the interpretation of the "2:00 transition to US Rule", and the fact that "2:00" is Wall-time, not S-time or U-time: I understand this to imply that a time of day value with a postfix "s" or "u" in the UNTIL column would preclude that it "just works". Now there are cases where two transitions (an increase in SAVE value after a decrease in STDOFF) are colaesced, and where the corresponding UNTIL column has a time of day value with a postfixed "s"; one example is for Kaliningrad. My comment asks how your code deals with such cases, given that it would not "just work". I hope this makes my point clear(er). Michael Deckers.
Got it, thanks for the clarification, it helps to have more context about what you were thinking. I was focused on Grand_Turk, so probably simplified my exposition too much. I think the better explanation is that the 2:00 in the UNTIL field precisely matches the 2:00 in the AT in the US Rule, so there is only one transition, instead of 2 transitions. Same thing seems to happen with the 2:00s in the UNTIL for Kaliningrad, which matches the 2:00s in the AT for the Russia Rule. The thing that I was wondering was whether the semantics of the zone info text files actually mandated a transition from 2:00 to 1:00 for Grand_Turk, if the Zone entry was set to 2:00, instead of its current 3:00 value. Or whether the special case in the zic compiler was an additional rule on top of the normal semantics of the zone info text files, in order to parse the Grand_Turk case properly. I managed to convince myself that the normal rules of the text files were sufficient for Grand_Turk. And my parsing code seems to implement those normal rules properly. On Tue, Dec 29, 2020 at 4:11 PM Michael H Deckers < michael.h.deckers@googlemail.com> wrote:
On 2020-12-29 22:45, Brian Park wrote:
Then I wish you had said exactly that, instead of having me go off on a tangent.
So do I. I am sorry to have lead you astray.
Maybe I am misinterpreting something, but I don't understand why you are being confrontational about this. I am presenting results from 2 different TZ parsers, using algorithms developed independently. I thought this would be interesting and relevant to this topic.
I do not want to sound confrontational but I do want to be short. My comment on your original post concerned only one sentence, and I did in no way comment on the rest of that post.
I don't understand the point that you are trying to make.
The complete sentence I commented about is:
The core of why it "just works" seems be in the interpretation of the "2:00 transition to US Rule", and the fact that "2:00" is Wall-time, not S-time or U-time:
I understand this to imply that a time of day value with a postfix "s" or "u" in the UNTIL column would preclude that it "just works".
Now there are cases where two transitions (an increase in SAVE value after a decrease in STDOFF) are colaesced, and where the corresponding UNTIL column has a time of day value with a postfixed "s"; one example is for Kaliningrad.
My comment asks how your code deals with such cases, given that it would not "just work".
I hope this makes my point clear(er).
Michael Deckers.
On 12/31/20 11:22 AM, Brian Park wrote:
The thing that I was wondering was whether the semantics of the zone info text files actually mandated a transition from 2:00 to 1:00 for Grand_Turk, if the Zone entry was set to 2:00, instead of its current 3:00 value. Or whether the special case in the zic compiler was an additional rule on top of the normal semantics of the zone info text files, in order to parse the Grand_Turk case properly.
The latter. That is, original semantics would have required two transitions there, but when this sort of problem was discovered in the mid-1990s the zic behavior was changed by adding the special case. Unfortunately, this special case wasn't documented in the zic man page until recently. The recent doc change is here: https://github.com/eggert/tz/commit/b804a741d5cdb5d1b761de9ad053ba916fa08636 and you can see a recent message about this here: https://mm.icann.org/pipermail/tz/2020-December/029616.html
On Thu, Dec 31, 2020 at 1:14 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
On 12/31/20 11:22 AM, Brian Park wrote:
The thing that I was wondering was whether the semantics of the zone info text files actually mandated a transition from 2:00 to 1:00 for Grand_Turk, if the Zone entry was set to 2:00, instead of its current 3:00 value. Or whether the special case in the zic compiler was an additional rule on top of the normal semantics of the zone info text files, in order to parse the Grand_Turk case properly.
The latter. That is, original semantics would have required two transitions there, but when this sort of problem was discovered in the mid-1990s the zic behavior was changed by adding the special case. Unfortunately, this special case wasn't documented in the zic man page until recently.
The recent doc change is here:
https://github.com/eggert/tz/commit/b804a741d5cdb5d1b761de9ad053ba916fa08636
and you can see a recent message about this here:
Yup, I did read those messages, and the 1996 emails too. I am still confused why the normal semantics require 2 transitions. Could you be kind enough to explain? Because I implemented my code before I knew about the special code in zic(1) to handle this case. My code produces only one transition for Grand_Turk at 2:00, and one transition for the original Atlantic/Stanley zone at 00:00 on Sept 15 1985, as referenced in the 1996 email thread. Brian
On 2020-12-05 19:45, Tim Parenti wrote:
Trying out "2:30" in the UNTIL field should mean that the Zone transition should happen 30 minutes BEFORE the US Rule would take effect (and thus require two transitions), but by wall-clock time, it would "appear" to be 30 minutes AFTER (fooling zic into thinking only the "latter" Zone transition is needed as the Rule would have "already" taken effect). And, indeed, the results are similarly incorrect here, too:....
If the behavior of zic is as I guess it is, a value of 2018 Mar 11 1:59:59 in the UNTO field will cause two jumps within 1 h + 1 s. Michael Deckers.
participants (5)
-
Arthur David Olson -
Brian Park -
Michael H Deckers -
Paul Eggert -
Tim Parenti