Unexpected last transition in Africa/Casablanca
Hi Paul, Here are the last transitions in Africa/Casablanca output (rearguard format): transition,type,[UTC time],[Type offset],[Type isDST] 3672612000,1,2086-05-19T02:00:00Z,PT1H,DST 3699828000,2,2087-03-30T02:00:00Z,PT0S,STD 3703456800,3,2087-05-11T02:00:00Z,PT1H,STD So change is from standard to standard, even though offset changes from 0h to 1h. Is that expected behaviour? As Africa/El_Aaiun is using Morocco rules, its transitions are also affected. This affects 64-bit section only, 32-bit section looks fine to me. Thanks, Almaz Data above is output of debug tools used in Android. The way I've built data an byte manipulations if needed: mkdir /tmp/zic cd /tmp/zic git clone https://github.com/eggert/tz.git . make -C . zic make -C . NDATA= rearguard.zi mkdir data ./zic -b fat -d ./data rearguard.zi byte manipulations: Let's skip 32-bit header: xxd -seek 20 -c 4 -l 24 data/Africa/Casablanca typecnt: 94 timecnt: 5 charcnt: 12 32-bit block size = 94 * 4 + 94 + 5 * 6 + 12 = 512 64-bit start = 44 + 512 = 556 64-bit header: xxd -seek 576 -c 4 -l 24 data/Africa/Casablanca: timecnt: 196 typecnt: 4 <- 32-bit section has 5 charcnt: 12 transition types offset: 556 + 44 + 196 * 8 = 2168 xxd -seek 2168 -c 1 -l 196 data/Africa/Casablanca ... 00000934: 02 . 00000935: 01 . 00000936: 02 . 00000937: 01 . 00000938: 02 . 00000939: 01 . 0000093a: 02 . 0000093b: 03 . <- the last transition local time records offset = 556 + 44 + 196 * 9 = 2364 xxd -seek 2364 -c 6 -l 24 data/Africa/Casablanca 0000093c: ffff f8e4 0000 ...... 00000942: 0000 0e10 0104 ...... 00000948: 0000 0000 0008 ...... <- standard time 0000094e: 0000 0e10 0004 ...... <- also standard time
On 3/22/22 05:40, Almaz Mingaleev wrote:
Here are the last transitions in Africa/Casablanca output (rearguard format):
transition,type,[UTC time],[Type offset],[Type isDST] 3672612000,1,2086-05-19T02:00:00Z,PT1H,DST 3699828000,2,2087-03-30T02:00:00Z,PT0S,STD 3703456800,3,2087-05-11T02:00:00Z,PT1H,STD
So change is from standard to standard, even though offset changes from 0h to 1h. Is that expected behaviour?
Yes, as tzdb frowns on permanent daylight saving time for reasons discussed recently. This issue occurs due to the rearguard format's contortion of post-2018 Morocco data to pretend that standard time is daylight saving and vice versa. Even with that contortion, we don't want the normal post-2087 prediction (standard time only) to be contorted into a rearguard post-2087 prediction of daylight saving only.
I see, thanks. Unfortunately such standard-to-standard time transitions break DST offset finding logic on Android (code is here [1]). For now I've reverted [2] (ironically the problem was reported by me). Is there a less intrusive way to keep the old behaviour? [1] https://android.googlesource.com/platform/external/icu/+/master/android_icu4... [2] https://github.com/eggert/tz/commit/cec7d9e2e83f8a3faa2367e0d45383a1557889ed On Tue, 22 Mar 2022 at 15:15, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 3/22/22 05:40, Almaz Mingaleev wrote:
Here are the last transitions in Africa/Casablanca output (rearguard format):
transition,type,[UTC time],[Type offset],[Type isDST] 3672612000 <(367)%20261-2000>,1,2086-05-19T02:00:00Z,PT1H,DST 3699828000,2,2087-03-30T02:00:00Z,PT0S,STD 3703456800,3,2087-05-11T02:00:00Z,PT1H,STD
So change is from standard to standard, even though offset changes from 0h to 1h. Is that expected behaviour?
Yes, as tzdb frowns on permanent daylight saving time for reasons discussed recently.
This issue occurs due to the rearguard format's contortion of post-2018 Morocco data to pretend that standard time is daylight saving and vice versa. Even with that contortion, we don't want the normal post-2087 prediction (standard time only) to be contorted into a rearguard post-2087 prediction of daylight saving only.
On 3/28/22 09:03, Almaz Mingaleev wrote:
Unfortunately such standard-to-standard time transitions break DST offset finding logic on Android (code is here [1]).
Could you explain the problem more? There are lots of places where timezones switch from one standard time to another, and evidently these other transitions don't cause a problem. What's different here? Here's the output of 'zdump -i -c 2085,2100 Africa/Casablanca' if you're using rearguard format:
TZ="Africa/Casablanca" - - +01 1 2085-04-22 02 +00 2085-06-03 03 +01 1 2086-04-14 02 +00 2086-05-19 03 +01 1 2087-03-30 02 +00 2087-05-11 03 +01
If we want the last line to be standard time, what should the previous lines look like if we want to avoid the Android transition-finding problem? For example, would it be OK if the penultimate line were absent?
I suspect it's a similar prime On March 28, 2022 5:33:36 PM UTC, Paul Eggert via tz <tz@iana.org> wrote:
On 3/28/22 09:03, Almaz Mingaleev wrote:
Unfortunately such standard-to-standard time transitions break DST offset finding logic on Android (code is here [1]).
Could you explain the problem more? There are lots of places where timezones switch from one standard time to another, and evidently these other transitions don't cause a problem. What's different here?
Here's the output of 'zdump -i -c 2085,2100 Africa/Casablanca' if you're using rearguard format:
TZ="Africa/Casablanca" - - +01 1 2085-04-22 02 +00 2085-06-03 03 +01 1 2086-04-14 02 +00 2086-05-19 03 +01 1 2087-03-30 02 +00 2087-05-11 03 +01
If we want the last line to be standard time, what should the previous lines look like if we want to avoid the Android transition-finding problem? For example, would it be OK if the penultimate line were absent?
There are APIs like TimeZone.getDSTSavings [1]. As of now we calculate it as the difference between the last DST and standard offsets in the transition table. As the last transition is marked as standard these offsets are equal, and the method returns 0, which is wrong.
would it be OK if the penultimate line were absent? Removing the last transition would help.
[1] https://docs.oracle.com/javase/8/docs/api/java/util/TimeZone.html#getDSTSavi... On Mon, 28 Mar 2022 at 18:33, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 3/28/22 09:03, Almaz Mingaleev wrote:
Unfortunately such standard-to-standard time transitions break DST offset finding logic on Android (code is here [1]).
Could you explain the problem more? There are lots of places where timezones switch from one standard time to another, and evidently these other transitions don't cause a problem. What's different here?
Here's the output of 'zdump -i -c 2085,2100 Africa/Casablanca' if you're using rearguard format:
TZ="Africa/Casablanca" - - +01 1 2085-04-22 02 +00 2085-06-03 03 +01 1 2086-04-14 02 +00 2086-05-19 03 +01 1 2087-03-30 02 +00 2087-05-11 03 +01
If we want the last line to be standard time, what should the previous lines look like if we want to avoid the Android transition-finding problem? For example, would it be OK if the penultimate line were absent?
On 3/29/22 09:33, Almaz Mingaleev wrote:
Removing the last transition would help.
OK. However, doing that would cause the rearguard and main timestamps to disagree in UTC offset for a month or so. It'd be less intrusive to add a transition after the last transition, which I hope would also work around the problem (as it'd exhibit a similar pattern) but would cause the rearguard and main timestamps to disagree only with respect to the isdst flag for a day, which is less of a difference. I installed the attached proposed patch to do that. Does it work for you? If not, perhaps we should revert all these workarounds, and go back to what we were doing in this area before commit cec7d9e2e83f8a3faa2367e0d45383a1557889ed dated 2022-01-10 11:31:54 -08.
On Mar 30, 2022, at 11:36 AM, Paul Eggert via tz <tz@iana.org> wrote:
It'd be less intrusive to add a transition after the last transition, which I hope would also work around the problem (as it'd exhibit a similar pattern) but would cause the rearguard and main timestamps to disagree only with respect to the isdst flag for a day, which is less of a difference.
Oh, it's this flag again. A C22 draft as of October 18, 2021: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2731.pdf says: 7.27.1 Components of time The header <time.h> defines several macros, and declares types and functions for manipulating time. Many functions deal with a calendar time that represents the current date (according to the Gregorian calendar) and time. Some functions deal with local time, which is the calendar time expressed for some specific time zone, and with Daylight Saving Time, which is a temporary change in the algorithm for determining local time. The local time zone and Daylight Saving Time are implementation-defined. ... The value of tm_isdst is positive if Daylight Saving Time is in effect, zero if Daylight Saving Time is not in effect, and negative if the information is not available. As I read it: an implementation that interprets "Daylight Saving Time" as "the clocks are temporarily turned forward", even if, as in Ireland, turning the clock forward is a change to *standard* time, would be valid, as it's a temporary change - as is the change back to "winter time"; an implementation that interprets "Daylight Saving Time" as "the clocks are temporarily changed from standard time", meaning that, in Ireland, "Daylight Saving Time" would be "winter time" would also be valid, as it's again a temporary change. On the other hand, an implementation that interprets "Daylight Saving Time" as "the clocks are temporarily turned *backward*" would also be valid, as long as they're temporarily turned forward as well. That ambiguity would require that the standard address what "*the* algorithm for determining local time", emphasis on "*the*" as in "this one is Official", is - is it the algorithm used during what the applicable government deems to be "standard time", or what? "The local time zone and Daylight Saving Time are implementation-defined." sounds as if they're trying to avoid that particular tarpit. However, a change to "permanent Daylight Saving Time" wouldn't be "Daylight Saving Time", as the change wouldn't be permanent. Should somebody ask the C standards committee to clarify this?
On Mar 30, 2022, at 16:14, Guy Harris via tz <tz@iana.org> wrote:
Oh, it's this flag again.
A C22 draft as of October 18, 2021:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2731.pdf <http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2731.pdf>
In light of the apparent ambiguities, might the best option be to return a -1 for all regions that observe “dst” (howsoever defined) during any part of the year? Cheers! |---------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |---------------------------------------------------------------------| | A room without books is like a body without a soul. | | | | -- Cicero | |---------------------------------------------------------------------|
Hi Paul, Sorry for the delay. Unfortunately changing isdst flag only won't help here - in this case the last DST and STD transition offsets are equal, which breaks DST savings finding logic - it would return 0, as in my first email, where last DST and STD offsets are 1 hour. On Wed, 30 Mar 2022 at 19:36, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 3/29/22 09:33, Almaz Mingaleev wrote:
Removing the last transition would help.
OK. However, doing that would cause the rearguard and main timestamps to disagree in UTC offset for a month or so. It'd be less intrusive to add a transition after the last transition, which I hope would also work around the problem (as it'd exhibit a similar pattern) but would cause the rearguard and main timestamps to disagree only with respect to the isdst flag for a day, which is less of a difference.
I installed the attached proposed patch to do that. Does it work for you? If not, perhaps we should revert all these workarounds, and go back to what we were doing in this area before commit cec7d9e2e83f8a3faa2367e0d45383a1557889ed dated 2022-01-10 11:31:54 -08.
On 5/18/22 02:32, Almaz Mingaleev wrote:
Unfortunately changing isdst flag only won't help here - in this case the last DST and STD transition offsets are equal, which breaks DST savings finding logic - it would return 0, as in my first email, where last DST and STD offsets are 1 hour.
On Wed, 30 Mar 2022 at 19:36, Paul Eggert <eggert@cs.ucla.edu> wrote:...
perhaps we should revert all these workarounds, and go back to what we were doing in this area before commit cec7d9e2e83f8a3faa2367e0d45383a1557889ed dated 2022-01-10 11:31:54 -08.
OK, in that case let's revert as mentioned above. I installed the attached proposed patch into the development repository. It is a bit disconcerting to see comments like the following <https://android.googlesource.com/platform/external/icu/+/master/android_icu4...>:
// This test means that for somewhere like Morocco, which tried DST in 2009 but has // no future plans (and thus no future schedule info) will report "true" from // useDaylightTime at the start of 2009 but "false" at the end. This seems appropriate
which indicates that the code (or at least its commentary) doesn't match reality, as Morocco obviously is using DST now (whichever way you model it). It's not good that TZDB now has yet another rearguard hack to work around the code's mismatch with reality. What would it take to get this fixed properly in Android and ICU and Java, and to support negative DST and all that? I know this can't be fixed overnight, but what would it take for a fix in the long run? As I understand it, Java currently can't even handle POSIX TZ strings much less TZDB data, and surely this is undesirable even aside from TZDB.
Thanks for fix! Can't really tell about Java, haven't spoken with OpenJDK folks. The last time I've asked ICU about negative DST offsets they had no plans to support them. On the Android side we have a list of things that can go wrong, but given ICU situation no work has been done in that direction yet.
Java currently can't even handle POSIX TZ strings much Old APIs (java.util.TimeZone) definitely can't handle DST only time zones, but I _think_ that new APIs* are in better shape and can handle POSIX TZ strings.
* Well, Java 8 was released in 2014, maybe not that new On Wed, 18 May 2022 at 18:41, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 5/18/22 02:32, Almaz Mingaleev wrote:
Unfortunately changing isdst flag only won't help here - in this case the last DST and STD transition offsets are equal, which breaks DST savings finding logic - it would return 0, as in my first email, where last DST and STD offsets are 1 hour.
On Wed, 30 Mar 2022 at 19:36, Paul Eggert <eggert@cs.ucla.edu> wrote:...
perhaps we should revert all these workarounds, and go back to what we were doing in this area before commit cec7d9e2e83f8a3faa2367e0d45383a1557889ed dated 2022-01-10 11:31:54 -08.
OK, in that case let's revert as mentioned above. I installed the attached proposed patch into the development repository.
It is a bit disconcerting to see comments like the following < https://android.googlesource.com/platform/external/icu/+/master/android_icu4...
:
// This test means that for somewhere like Morocco, which
tried DST in 2009 but has
// no future plans (and thus no future schedule info) will
report "true" from
// useDaylightTime at the start of 2009 but "false" at the
end. This seems appropriate
which indicates that the code (or at least its commentary) doesn't match reality, as Morocco obviously is using DST now (whichever way you model it). It's not good that TZDB now has yet another rearguard hack to work around the code's mismatch with reality.
What would it take to get this fixed properly in Android and ICU and Java, and to support negative DST and all that? I know this can't be fixed overnight, but what would it take for a fix in the long run? As I understand it, Java currently can't even handle POSIX TZ strings much less TZDB data, and surely this is undesirable even aside from TZDB.
On Wed, 18 May 2022 at 18:41, Paul Eggert via tz <tz@iana.org> wrote:
What would it take to get this fixed properly in Android and ICU and Java, and to support negative DST and all that? I know this can't be fixed overnight, but what would it take for a fix in the long run?
It will never be "fixed". Because from our perspective TZDB is broken, not Java/Android/ICU. Key here is that Java (and thus indirectly Android) and ICU take backwards compatibility more seriously than TZDB. The DST flag has a specific meaning to Java, and there is no interest in whether a location has positively or negatively defined DST rules. For global-tz, I simply publish the standard and rearguard versions for consumers to pick from. https://github.com/JodaOrg/global-tz/releases/tag/2022agtz Perhaps TZDB could remove negative DST from the data and just refer to it only in comments? Maybe that way you could remove rearguard. Stephen
On 5/19/22 05:54, Stephen Colebourne wrote:
Key here is that Java (and thus indirectly Android) and ICU take backwards compatibility more seriously than TZDB.
TZDB is upward compatible with POSIX, whereas from what others are saying it appears that Java's old API is not compatible because the implementation doesn't support negative DST. This doesn't sound like Java and ICU treat "backwards compatibility more seriously"; on the contrary, it sounds like the Java implementation is gratuitously incompatible with a standard that predates Java by several years. Almaz writes that Java's newer APIs can handle POSIX TZ strings so it appears Java's newer APIs are in better shape, which would be a good thing. Though I'm still puzzled as to why Java's old API can't support negative DST so as to be compatible with POSIX. A java.util.Timezone object's getDSTSavings method could return a negative integer just as easily as a positive one. Sure, some old code might break if getDSTSavings returns any value other than the usual 3600000 milliseconds, but the same old code might break if getDSTSavings returns 1800000 rather than 3600000 (which is something it *can* do now) so it shouldn't be a big deal for it to return -3600000 either.
participants (6)
-
Almaz Mingaleev -
Fred Gleason -
Guy Harris -
Paul Eggert -
Paul Ganssle -
Stephen Colebourne