tzcode-2014c breaks applications that use the binary tz files
Hi, After updating to 2014c, we've received a couple of reports of applications that started to misbehave (show wrong times). One example is the clock applet from Gnome3 (the one in the panel) which, on my machine, shows the time with a -7 minutes offset. Reverting these recent changes from zic(1) resolves the problem: 7fb077a9f zic: improve handling of low-valued time stamps 2586e998c zic: revert Macquarie-specific changes Leonardo
On 05/23/2014 02:27 PM, Leonardo Chiquitto wrote:
After updating to 2014c, we've received a couple of reports of applications that started to misbehave (show wrong times). One example is the clock applet from Gnome3
Thank you for the heads-up. Do all the broken applications use Glib (the Gnome library)? If so, thisappears to be Gnome bug 730332, which you can view here: https://bugzilla.gnome.org/show_bug.cgi?id=730332 Starting with 2014c, zic generates a transition at the minimum time value -2**63, to avoid ambiguities about what to do before then. The interval_end function of Glib's gtimezone.c subtracts one from this, to find the end time of the zeroth interval (i.e., the interval containing all the "early" time stamps); in interval_end this subtraction overflows and wraps around to 2**63 - 1, which causes Glib to go off the rails and assume that *all* time stamps are "early". For example, Glib computes Sao Paulo time stamps as if Brazil's circa-1913 rules were still in effect, i.e., as if Sao Paulo were at UTC-3:06:28, which is incorrect by 6 minutes 28 seconds, which is the error you're observing. This is a serious incompatibility. Distributions that use Glib should not use tz 2014c's code to generate binary files. The 2014c data files are fine; it's the 2014c zic.c that's causing the incompatibility. This is clearly a bug in Glib, and I assume it'll be fixed soon, but that's small consolation to the people affected by it. And one can't help but wonder which other software packages have similar bugs. A simple workaround is to back out the two zic.c changes you mentioned. Another possibility would be for zic.c to generate a transition at time -2**63 + 1 instead, so that the Glib code won't wrap around, but that sounds fragile. Yet another possibility might be to change zic.c to set the initial transition to be -2**60 instead of -2**63; this would comfortably predate the Big Bang so it's unlikely we'll see any real time stamps before then. I'll CC: this to Arthur David Olson to see whether he has a better idea.
Using something such as -2**60 (with a relevant explanatory note) seems reasonable. Do we face a similar challenge in the 32-bit realm? If so, -2**31 + 1 might be what to use there. --ado On Fri, May 23, 2014 at 4:45 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 05/23/2014 02:27 PM, Leonardo Chiquitto wrote:
After updating to 2014c, we've received a couple of reports of applications that started to misbehave (show wrong times). One example is the clock applet from Gnome3
Thank you for the heads-up.
Do all the broken applications use Glib (the Gnome library)? If so, thisappears to be Gnome bug 730332, which you can view here:
https://bugzilla.gnome.org/show_bug.cgi?id=730332
Starting with 2014c, zic generates a transition at the minimum time value -2**63, to avoid ambiguities about what to do before then. The interval_end function of Glib's gtimezone.c subtracts one from this, to find the end time of the zeroth interval (i.e., the interval containing all the "early" time stamps); in interval_end this subtraction overflows and wraps around to 2**63 - 1, which causes Glib to go off the rails and assume that *all* time stamps are "early". For example, Glib computes Sao Paulo time stamps as if Brazil's circa-1913 rules were still in effect, i.e., as if Sao Paulo were at UTC-3:06:28, which is incorrect by 6 minutes 28 seconds, which is the error you're observing.
This is a serious incompatibility. Distributions that use Glib should not use tz 2014c's code to generate binary files. The 2014c data files are fine; it's the 2014c zic.c that's causing the incompatibility.
This is clearly a bug in Glib, and I assume it'll be fixed soon, but that's small consolation to the people affected by it. And one can't help but wonder which other software packages have similar bugs.
A simple workaround is to back out the two zic.c changes you mentioned. Another possibility would be for zic.c to generate a transition at time -2**63 + 1 instead, so that the Glib code won't wrap around, but that sounds fragile.
Yet another possibility might be to change zic.c to set the initial transition to be -2**60 instead of -2**63; this would comfortably predate the Big Bang so it's unlikely we'll see any real time stamps before then.
I'll CC: this to Arthur David Olson to see whether he has a better idea.
On 05/23/2014 01:51 PM, Arthur David Olson wrote:
Do we face a similar challenge in the 32-bit realm?
Not for Glib, no. It uses 64-bit signed integers internally. If this were a problem in the 32-bit realm, wouldn't we have already run into it? Some of tzdata's transitions are before 1901 and thus predate the 32-bit window, so I assume that for many years we've been approximating them with a transition at -2**31 -- is my assumption incorrect? One other thought, just for fun. In the 64-bit realm, we could use the timestamp of the Big Bang, as time stamps before then are physically suspect anyway. If we use POSIX time, our current best guess is that the Big Bang happened 13.82 billion years ago, which (if we use the POSIX origin of 1970) would make it -13,820,000,000 * 365.2425 * 86400, or -436117076640000000 -- a 59-bit number. A downside of using -436117076640000000 is that we'd need to update that number as we get better estimates of when the Big Bang occurred, but maybe we could consider this to be part of the fun. After all, if Unix had been developed on a 64-bit machine and had used the Big Bang for the time_t origin, a lot of bugs would have been avoided in the first place....
just +1 for using the big bang... because "time stamps before then are physically suspect anyway." :) On 05/23/2014 11:10 PM, Paul Eggert wrote:
On 05/23/2014 01:51 PM, Arthur David Olson wrote:
Do we face a similar challenge in the 32-bit realm?
Not for Glib, no. It uses 64-bit signed integers internally.
If this were a problem in the 32-bit realm, wouldn't we have already run into it? Some of tzdata's transitions are before 1901 and thus predate the 32-bit window, so I assume that for many years we've been approximating them with a transition at -2**31 -- is my assumption incorrect?
One other thought, just for fun. In the 64-bit realm, we could use the timestamp of the Big Bang, as time stamps before then are physically suspect anyway. If we use POSIX time, our current best guess is that the Big Bang happened 13.82 billion years ago, which (if we use the POSIX origin of 1970) would make it -13,820,000,000 * 365.2425 * 86400, or -436117076640000000 -- a 59-bit number. A downside of using -436117076640000000 is that we'd need to update that number as we get better estimates of when the Big Bang occurred, but maybe we could consider this to be part of the fun. After all, if Unix had been developed on a 64-bit machine and had used the Big Bang for the time_t origin, a lot of bugs would have been avoided in the first place....
gunther Vermeir wrote:
just +1 for using the big bang... because "time stamps before then are physically suspect anyway." :)
Yes, I love this one! Just yesterday I was talking to an undergraduate UCLA computer science student about cosmological inflation, and now I can go back to him and tell him I squashed a Brazilian time stamp bug with the Big Bang! I'm attaching a patch which I've pushed into the experimental repository. This patch uses a more-justifiable estimate for the time of the Big Bang, namely the Planck+WP+highL+BAO best fit. I've tested this with the 2014c database and verified that it doesn't change the zdump output. I've also tested that it works around the GNOME bug for America/Sao_Paulo on Fedora 20 x86-64 (which uses Glib 2.38). Comments welcome. I'd like to generate a new tz version soon, since the GNOME bug is reasonably painful.
I like this idea, may be we can lobby POSIX to chance also :) Steven Weinberg will become jealous. re, wh Am 24.05.2014 04:16, schrieb Paul Eggert:
gunther Vermeir wrote:
just +1 for using the big bang... because "time stamps before then are physically suspect anyway." :)
Yes, I love this one! Just yesterday I was talking to an undergraduate UCLA computer science student about cosmological inflation, and now I can go back to him and tell him I squashed a Brazilian time stamp bug with the Big Bang!
I'm attaching a patch which I've pushed into the experimental repository. This patch uses a more-justifiable estimate for the time of the Big Bang, namely the Planck+WP+highL+BAO best fit. I've tested this with the 2014c database and verified that it doesn't change the zdump output. I've also tested that it works around the GNOME bug for America/Sao_Paulo on Fedora 20 x86-64 (which uses Glib 2.38).
Comments welcome. I'd like to generate a new tz version soon, since the GNOME bug is reasonably painful.
On Friday, May 23 2014, "Paul Eggert" wrote to "gunther Vermeir, tz@iana.org" saying:
gunther Vermeir wrote:
just +1 for using the big bang... because "time stamps before then are physically suspect anyway." :)
Yes, I love this one! Just yesterday I was talking to an undergraduate UCLA computer science student about cosmological inflation, and now I can go back to him and tell him I squashed a Brazilian time stamp bug with the Big Bang!
I'm attaching a patch which I've pushed into the experimental repository. This patch uses a more-justifiable estimate for the time of the Big Bang, namely the Planck+WP+highL+BAO best fit. I've tested this with the 2014c database and verified that it doesn't change the zdump output. I've also tested that it works around the GNOME bug for America/Sao_Paulo on Fedora 20 x86-64 (which uses Glib 2.38).
Comments welcome. I'd like to generate a new tz version soon, since the GNOME bug is reasonably painful.
My only concern would be whether anything doing GNOME-style behavior then wants to pass the minimum timestamp to localtime() or the like, which will fail for timestamps repesenting years earlier than -2^31 - 1900. -- Jonathan Lennox lennox@cs.columbia.edu
lennox@cs.columbia.edu wrote:
My only concern would be whether anything doing GNOME-style behavior then wants to pass the minimum timestamp to localtime() or the like, which will fail for timestamps repesenting years earlier than -2^31 - 1900.
Thanks, that is a possible bug that I hadn't considered. Unfortunately that boundary's value is INT_MIN - 1900, and INT_MIN isn't necessarily -2**31. For example, on Unisys ClearPath enterprise servers INT_MIN is 1 - 2**39; because tz binary files are supposed to be usable even on these unusual platforms, hardwiring -2**31 - 1900 into them would cause an unwanted glitch at the 32-bit boundary on these platforms. Conversely, if we used the conservative boundary -32767 - 1900 (which the C standard says should be representable on all platforms), we would insert an unwanted glitch on nearly every platform. And even on the oddball platform where INT_MIN == -32767 there could be problems within a few hours of the boundary, due to software being confused about localtime succeeding but gmtime failing or vice versa. I think this possible bug is less likely than the GNOME bug, so I'm inclined try the Big Bang and see whether it breaks anything compared to 2014b. If not, we can declare victory; otherwise, we can look into adjusting the boundary according to whatever other library bug we run into. In private email, Arthur David Olson pointed out that zic would still let pre-Big-Bang leap seconds slip through; the attached further patch (which I've pushed to the experimental repository) fixes this.
On 2014-05-25 13:25, Paul Eggert wrote:
lennox@cs.columbia.edu wrote:
My only concern would be whether anything doing GNOME-style behavior then wants to pass the minimum timestamp to localtime() or the like, which will fail for timestamps repesenting years earlier than -2^31 - 1900.
Thanks, that is a possible bug that I hadn't considered. Unfortunately that boundary's value is INT_MIN - 1900, and INT_MIN isn't necessarily -2**31. For example, on Unisys ClearPath enterprise servers INT_MIN is 1 - 2**39; because tz binary files are supposed to be usable even on these unusual platforms, hardwiring -2**31 - 1900 into them would cause an unwanted glitch at the 32-bit boundary on these platforms.
Conversely, if we used the conservative boundary -32767 - 1900 (which the C standard says should be representable on all platforms), we would insert an unwanted glitch on nearly every platform. And even on the oddball platform where INT_MIN == -32767 there could be problems within a few hours of the boundary, due to software being confused about localtime succeeding but gmtime failing or vice versa.
I think this possible bug is less likely than the GNOME bug, so I'm inclined try the Big Bang and see whether it breaks anything compared to 2014b. If not, we can declare victory; otherwise, we can look into adjusting the boundary according to whatever other library bug we run into.
In private email, Arthur David Olson pointed out that zic would still let pre-Big-Bang leap seconds slip through; the attached further patch (which I've pushed to the experimental repository) fixes this.
Given that all tz dates are in the Gregorian calendar, the first date observed in that calendar Fri, 15 Oct, 1582 00:00:00 which is Unix time -12219292800 on 64 bit time_t systems should be sufficiently early and within the ctime limitation of 4 digit years? It will also never be adjusted, unlike the current Big Bang estimate. All dates prior to that date have to be interpreted as Julian dates, and 10 days subtracted because of the Gregorian date adjustment. Alternatively Thu, 14 Sep, 1752 00:00:00 Unix time -6857222400 could be used, as the last date on which a large number of countries adopted the Gregorian calendar, and is still well before any adoption of Standard times and zones. All dates prior to that date may be interpreted as Julian dates, and 11 days subtracted because of the later Gregorian date adjustment. That date is also consistent with the output of the Unix cal command, which drops Sep 3-13, whereas 1582 shows Oct with 31 days. -- Take care. Thanks, Brian Inglis
Brian Inglis wrote:
Given that all tz dates are in the Gregorian calendar
The tz code and database use the proleptic Gregorian calendar, which goes backwards in time before 1582, so we needn't do anything special with 1582 (or with 1867 for Alaskan timestamps, etc.). There's a similar issue with seconds. The current definition of the SI second wasn't clearly established until 1997, but we can and do extend it far backwards in time, as in the Big Bang estimate.
"PE" == Paul Eggert <eggert@cs.ucla.edu> writes:
PE> I think this possible bug is less likely than the GNOME bug, so I'm PE> inclined try the Big Bang and see whether it breaks anything compared PE> to 2014b. I'd go with -(2^60), though, as (you?) first suggested, rather than any estimate of the bang. The value should be constant and avoid anything which could lead to a future flame war should estimates change. Its also easier to read/spot in binary, octal or hex. Put another way, as long as it uses integers mod a power of two, use a power of two. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 0x997A9F17ED7DAEA6
James Cloos wrote:
The value should be constant and avoid anything which could lead to a future flame war should estimates change.
Good point; we try to avoid topics that might cause flame wars. - 2**59 should be safe, as it's wayyy outside the error bounds, so the attached patch (installed in the experimental github repository) goes with that.
"PE" == Paul Eggert <eggert@cs.ucla.edu> writes:
PE> the attached patch (installed in the experimental github repository) PE> goes with [-2^59]. 60 seems nicer due to its roundness, but -2^60 might be INT_MIN for languages, such as the lisps, which steal bits off integers for internal use. As such, and given the reasoning specified in the comments in the patch, +1 on 59. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 0x997A9F17ED7DAEA6
Just to use as a data point, our (private) implementation that reads the binary files also had this issue. I had to revert to 2014b to generating the binary data until I could follow up and make sure the code was working correctly with the newly introduced transition. -Andrew On Fri, May 23, 2014 at 4:45 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 05/23/2014 02:27 PM, Leonardo Chiquitto wrote:
After updating to 2014c, we've received a couple of reports of applications that started to misbehave (show wrong times). One example is the clock applet from Gnome3
Thank you for the heads-up.
Do all the broken applications use Glib (the Gnome library)? If so, thisappears to be Gnome bug 730332, which you can view here:
https://bugzilla.gnome.org/show_bug.cgi?id=730332
Starting with 2014c, zic generates a transition at the minimum time value -2**63, to avoid ambiguities about what to do before then. The interval_end function of Glib's gtimezone.c subtracts one from this, to find the end time of the zeroth interval (i.e., the interval containing all the "early" time stamps); in interval_end this subtraction overflows and wraps around to 2**63 - 1, which causes Glib to go off the rails and assume that *all* time stamps are "early". For example, Glib computes Sao Paulo time stamps as if Brazil's circa-1913 rules were still in effect, i.e., as if Sao Paulo were at UTC-3:06:28, which is incorrect by 6 minutes 28 seconds, which is the error you're observing.
This is a serious incompatibility. Distributions that use Glib should not use tz 2014c's code to generate binary files. The 2014c data files are fine; it's the 2014c zic.c that's causing the incompatibility.
This is clearly a bug in Glib, and I assume it'll be fixed soon, but that's small consolation to the people affected by it. And one can't help but wonder which other software packages have similar bugs.
A simple workaround is to back out the two zic.c changes you mentioned. Another possibility would be for zic.c to generate a transition at time -2**63 + 1 instead, so that the Glib code won't wrap around, but that sounds fragile.
Yet another possibility might be to change zic.c to set the initial transition to be -2**60 instead of -2**63; this would comfortably predate the Big Bang so it's unlikely we'll see any real time stamps before then.
I'll CC: this to Arthur David Olson to see whether he has a better idea.
On Fri, May 23, 2014 at 10:45 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
On 05/23/2014 02:27 PM, Leonardo Chiquitto wrote:
After updating to 2014c, we've received a couple of reports of applications that started to misbehave (show wrong times). One example is the clock applet from Gnome3
Thank you for the heads-up.
Do all the broken applications use Glib (the Gnome library)? If so, this appears to be Gnome bug 730332, which you can view here:
Indeed this seems to be the problem. One of the reports was about gdm and other Gnome applications, but we received at least one more from a different application that also made assumptions on the format of the binary files.
Starting with 2014c, zic generates a transition at the minimum time value -2**63, to avoid ambiguities about what to do before then. The interval_end function of Glib's gtimezone.c subtracts one from this, to find the end time of the zeroth interval (i.e., the interval containing all the "early" time stamps); in interval_end this subtraction overflows and wraps around to 2**63 - 1, which causes Glib to go off the rails and assume that *all* time stamps are "early". For example, Glib computes Sao Paulo time stamps as if Brazil's circa-1913 rules were still in effect, i.e., as if Sao Paulo were at UTC-3:06:28, which is incorrect by 6 minutes 28 seconds, which is the error you're observing.
Thanks for the detailed background information!
This is a serious incompatibility. Distributions that use Glib should not use tz 2014c's code to generate binary files. The 2014c data files are fine; it's the 2014c zic.c that's causing the incompatibility.
This is clearly a bug in Glib, and I assume it'll be fixed soon, but that's small consolation to the people affected by it. And one can't help but wonder which other software packages have similar bugs.
Yes, that's the problem. Specially when we consider that there are probably proprietary software out there doing the same.
A simple workaround is to back out the two zic.c changes you mentioned.
For now, this is what we're going to do as quickly as possible.
Another possibility would be for zic.c to generate a transition at time -2**63 + 1 instead, so that the Glib code won't wrap around, but that sounds fragile.
Yet another possibility might be to change zic.c to set the initial transition to be -2**60 instead of -2**63; this would comfortably predate the Big Bang so it's unlikely we'll see any real time stamps before then.
I'll CC: this to Arthur David Olson to see whether he has a better idea.
Thanks for your fast response and have a nice weekend, Leonardo
participants (9)
-
Andrew Paprocki -
Arthur David Olson -
Brian Inglis -
gunther Vermeir -
James Cloos -
lennox＠cs.columbia.edu -
Leonardo Chiquitto -
Paul Eggert -
walter harms