Merged 1970+ time zones should always return -1 pre-1970
I think for all zones with data merged prior to 1970, the data and API should always return *-1* for all times prior to 1970-01-01 00:00:00Z, as anything else is inaccurate or a kludge. Arbitrary disposition of any valid historical data is itself really political, for ease of administration, not really technical. I firmly believe that no decision on the list has ever been made on a basis of in-/equity or un-/fairness, as those would be political decisions, and we have bent over backwards for years to avoid any such appearance never mind actuality, devoting probably the majority of the volume of posts and the majority of the volume of words to such discussions. I have always taken the position that any churn not necessitated by changes on the ground (or government offices) is detrimental to the project and the database, and places a massive burden on downstreams who make good use of that data, including the CLDR and ICU projects, astronomy and astrology applications, all facets of the transportation industry, the databases they maintain, and the storage of what was the truth at a previous point in time, and future application uses of that previously stored data. Any such churn should be mediated by the request the list makes of time zone decision makers: to provide adequate notice of such changes, at least six months, preferably longer. I can only believe that the adminstrators of this database are under some form of political pressure (perhaps /only/ "vitriol in private email", which appears to be nearing the norm in certain arenas) to dispose of some data to achieve some misguided form of "equity" and "fairness", in response to some misguided perception of "inequity" and "unfairness" that is not apparent to members of this mailing list, and from the reactions, nor does it appear to have been adequately or clearly expressed in a form that members of this list can readily comprehend, nor is it clearly exactly when these political terms entered the discussion, nor what business they have being in it. As in any project where decisions are questionable, it is necessary to back off making any! The proposers of drastic changes need to enunciate clearly: * exactly what technical problem they perceive, and why it appears to be a problem; * some alternative options for technical solutions and/or approaches to address the technical problem; * how each alternative option for technical solutions and/or approaches to address the problem will satisfy the majority of needs of the majority of the stakeholders, and impacts on those unsatisfied; * why they think the option they propose to adopt is the best and most comprehensive solution to address the problem, and satisfies the most needs of the most stakeholders. This is standard practice in most commercial projects. It is often used when stakeholders can not quickly make a decision, decide on policy, or easily change policy. It allows development to proceed in parallel on all the offered options. When stakeholders finally decide or act, perhaps months along the timeframe nearing release, the selected option can be preferentially enabled, possibly with only minor tweaks or just data changes, and quickly regression tested to ensure the effect is precisely as desired. Of course, the best decision to satisfy the most needs of the most stakeholders is often the decision to just say "NO !" ;^> -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.]
On Tue, 28 Sep 2021, Brian Inglis via tz wrote:
I think for all zones with data merged prior to 1970, the data and API should always return *-1* for all times prior to 1970-01-01 00:00:00Z, as anything else is inaccurate or a kludge.
That is only an option if you use the code bundled with the data. I can't simply change PHP to throw a "can't have this Date" exception for pre-1970 timestamps. And removing/changing the already existing timezone information to either that of a different country or UTC, is also not practical. It is a backwards compatibiliy break that I *will* get emails about. cheers, Derick -- PHP 7.4 Release Manager Host of PHP Internals News: https://phpinternals.news Like Xdebug? Consider supporting me: https://xdebug.org/support https://derickrethans.nl | https://xdebug.org | https://dram.io twitter: @derickr and @xdebug
On 9/27/21 11:32 PM, Brian Inglis via tz wrote:
I think for all zones with data merged prior to 1970, the data and API should always return *-1* for all times prior to 1970-01-01 00:00:00Z, as anything else is inaccurate or a kludge.
Although we could do that as an option, using that option would cause more data churn than other proposals presented. Many Zones in tzdb are "zones with data merged prior to 1970". America/New_York, for example, merges many regions that did not observe the same time before 1970. The same is true of America/Chicago, Europe/Paris, Asia/Shanghai, etc. For most tzdb uses, I would say this merging is far more the rule than the exception.
On 2021-09-28 16:00, Paul Eggert via tz wrote:
Many Zones in tzdb are "zones with data merged prior to 1970". America/New_York, for example, merges many regions that did not observe the same time before 1970.
Yes, but it merged only zones that _never_ had an entry in tzdb. What we are doing now is merging tome zones that have existed separately for years within tzdb. That's a completely different matter. Micheal Deckers.
On 9/28/21 11:59, Michael H Deckers wrote:
America/New_York, for example, merges many regions that did not observe the same time before 1970.
Yes, but it merged only zones that _never_ had an entry in tzdb. What we are doing now is merging tome zones that have existed separately for years within tzdb.
There are many examples of those merges too. For example, Europe/Zurich currently covers Liechtenstein even though before 2013 there was a separate Zone for Europe/Vaduz. And anyway, saying "we've always done it that way" would not be an adequate response to valid fairness concerns.
On 2021-09-28 19:13, Paul Eggert wrote:
There are many examples of those merges too. For example, Europe/Zurich currently covers Liechtenstein even though before 2013 there was a separate Zone for Europe/Vaduz.
And anyway, saying "we've always done it that way" would not be an adequate response to valid fairness concerns.
I rather see it the other way around: any merging is potentially unfair. When you merged Africa/Bamako with Africa/Abidjan (in 2014f), you also eradicated for the people of Mali their date of independence from France (1960-06-20), when they switched to UTC + 00 h. It certainly looks as if the colonial times were made invisible; and while this may be unintentional, it is an impression that must be avoided by all means, in my opinion, even at the cost of some more Bytes. Michael Deckers.
On 9/28/21 13:27, Michael H Deckers wrote:
When you merged Africa/Bamako with Africa/Abidjan (in 2014f), you also eradicated for the people of Mali their date of independence from France
Nothing was "eradicated". All that information is still in tzdb. Also, the 'backzone' line you're referring to ("-1:00 - -01 1960 Jun 20" [1]) is surely wrong, and so doesn't belong in a high-quality timekeeping database regardless of when Mali became independent. The more we get bogged down with irrelevant 'backzone' trivia like the date of Mali's independence, the less time we'll have to make genuinely useful improvements to tzdb, such as some of the improvements recently discussed on this list. Let's focus on these improvements rather than rehashing past mistakes. [1] https://github.com/eggert/tz/blob/6c15079a0fa34a879639e0f9baab92a472ef63a0/b...
On Tue, 28 Sep 2021, Paul Eggert via tz wrote:
On 9/28/21 13:27, Michael H Deckers wrote:
When you merged Africa/Bamako with Africa/Abidjan (in 2014f), you also eradicated for the people of Mali their date of independence from France
Nothing was "eradicated". All that information is still in tzdb.
It might be in tzdb, but it was not distributed by most, if not all, Linux/*BSD distributions. Which for the end users effectively means that it was eradicated. cheers, Derick
Date: Tue, 28 Sep 2021 16:38:30 -0700 From: Paul Eggert via tz <tz@iana.org> Message-ID: <086e075f-05fc-4f4f-a2ff-c4eeb2f2ab2f@cs.ucla.edu> | Also, the 'backzone' line you're referring to ("-1:00 - -01 1960 Jun 20" | [1]) is surely wrong, Do you have some evidence for that? Are you claiming that (what is now) Mali used something other than -1:00 from 1934 to 1960? Or just that the precise date/time of the change "cannot" be correct? If it is the latter, then without evidence of the change happening at some other time, I would tend to believe that as it is written. Independence had been agreed with France months earlier - that is, they had plenty of time to prepare, and was to happen (and happened) on that date. Assuming that part of the preparation was to switch timezones along with independence, which is not an absurd suggestion, then that rule seems entirely plausible to me. | and so doesn't belong in a high-quality | timekeeping database regardless of when Mali became independent. And you believe that a high-quality timekeeping database when converting times from (what is now) Mali in 1950 should produce answers that are an hour incorrect? If the answer to that is "it is before 1970, so we promise nothing" then that same answer would also apply to the earlier rule, as it was, even if wrong, but then while there might have been a few hours, or even days or weeks perhaps, when the answer was wrong, surely that's better than 26 years of erroneous answers? | The more we get bogged down with irrelevant 'backzone' trivia like the | date of Mali's independence, There's no need to get bogged down with that, I don't believe that is disputed. In cases where independence came via revolution, then the exact time it happened could be disputed. Not here. And who or what has been bogging us down with that discussion? I've just done a search of my copy of this list, and the only mentions of Mali that appear at all, are in Michael H Deckers recent message, and then your reply (and then there will be this message as well, eventually - maybe some more recent messages in the past hour or two or following this), and locations where Mali has appeared in tzdata in a patch (either as its data was being amended somehow, or when some other nearby data was and it was caught in the context diff). (My copy of list data goes back to May 1988...) | the less time we'll have to make genuinely useful improvements to tzdb, | such as some of the improvements recently discussed on this list. I must have missed something, but there have been lots of recent messages, so that is possible but I have no idea what improvements you mean. Some data corrections, sure, and many of those pre 1970, and while those are likely improvements, I cannot see how they're any more important. Aside from those, I don't recall anything recent that is really any kind of notable improvement. | Let's focus on these improvements rather than rehashing past mistakes. One improvement needed is to fix the past mistakes - like that one. "We got it wrong, sigh, never mind, what's next" isn't a good approach. kre ps: why does Pacific/Honolulu still exist as a zone given the apparent policy? As best I can tell it is identical to Etc/GMT+10 for all dates after 1970, so shouldn't it also be moved to backzone and made into a link? Further a comment just above its rules says: # We're left to guess the time of day when Act 163 was approved; guess noon. and the rule incorporates that guess. That's far less likely to be correct than that Mali's timezone changed at midnight on the day of independence. But no, I am not suggesting that actually happen - far from it - but if the requirement for fairness was to eliminate everything that isn't needed for post 1970 accuracy, then surely that is required. Of course I don't think I'd like to be a TZ coordinator in the US if the US press discovered a plan to wipe Hawaii off the map.
On 9/29/21 9:32 AM, Robert Elz wrote:
Date: Tue, 28 Sep 2021 16:38:30 -0700 From: Paul Eggert via tz<tz@iana.org> Message-ID:<086e075f-05fc-4f4f-a2ff-c4eeb2f2ab2f@cs.ucla.edu>
| Also, the 'backzone' line you're referring to ("-1:00 - -01 1960 Jun 20" | [1]) is surely wrong,
... without evidence of the change happening at some other time, I would tend to believe that as it is written. Independence had been agreed with France months earlier - that is, they had plenty of time to prepare, and was to happen (and happened) on that date. Assuming that part of the preparation was to switch timezones along with independence, which is not an absurd suggestion
No, it is completely absurd. It's not how timezone rule changes work. No politician, no matter how amateur, would arrange to change the clocks merely because of Independence Day. It'd be a major political blunder if fiddling with the clocks is signaled as one of the most important things the new legislature or executive could do. And it would cause crowds to show up to parades at the wrong time. That particular data entry must be wrong, and I can say this as someone with extensive experience reviewing dubious timezone data (unfortunately this is not a large club).
| The more we get bogged down with irrelevant 'backzone' trivia like the | date of Mali's independence,
There's no need to get bogged down with that
Then let's not.
why does Pacific/Honolulu still exist as a zone given the apparent policy?
It differs from all other Zones in its combination of UT offsets and timezone abbreviations. My original proposal (now mostly reverted) involved writing a program that merged all Zones that agreed after 1970, so I doubt whether I missed any.
Date: Wed, 29 Sep 2021 10:08:04 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <701226da-b57c-1fa1-a33b-50522a0f6de0@cs.ucla.edu> | No, it is completely absurd. It's not how timezone rule changes work. No | politician, no matter how amateur, would arrange to change the clocks | merely because of Independence Day. I think that's a stretch. It is certainly one thing that can be changed and would indicate that "things are different now" before any of the harder and more costly changes could be implemented. Whether that happened or not I don't know - but I certainly don't simply write it off as absurd. | It'd be a major political blunder if | fiddling with the clocks is signaled as one of the most important things | the new legislature or executive could do. I have seen no evidence of a claim like that, I would not be surprised to learn there were other issues considered considerably more important. | That particular data entry must be wrong, Sorry, "must be" is not an argument that has any traction with me at all, no matter what the issue. Unless you can show it is impossible (which clearly can't be done here, as it obviously would have been possible, regardless of how likely it might be considered) then only evidence of what actually happened will do. Clearly, unless the assertion is that Mali was never in the -01:00 zone, there had to be a change sometime, as it now isn't. Even if it turns out that it was a guess, and is wrong, that is no worse than the guess in the Honolulu zone is it? Should that one be removed too? | It differs from all other Zones in its combination of UT offsets and | timezone abbreviations. Eliminating abbreviations is another task that's been undertaken with some abandon - Honolulu's could be changed to "-10" then it would (post 1970) be identical to Etc/GMT+10 and you'd have one less zone to manage. Furthermore, both those changes would surely be more fair and equitable? kre
Originally, such merges (geographic / Etc) were in the proposed mergers, but it was decided that this was a bad idea (since linking geographic -> Etc makes the counterintuitive signs of the Etc/GMT zones the "canonical" names of those zones, and linking Etc -> geographic breaks the [technically out-of-scope, but still useful] assumption that the Etc/GMT zones would always produce the same offset. See this email and preceding emails for the discussion that occurred: https://mm.icann.org/pipermail/tz/2021-May/030136.html Best, Emily Crandall Fleischman On Wed, 29 Sept 2021 at 15:07, Robert Elz via tz <tz@iana.org> wrote:
Date: Wed, 29 Sep 2021 10:08:04 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <701226da-b57c-1fa1-a33b-50522a0f6de0@cs.ucla.edu>
| No, it is completely absurd. It's not how timezone rule changes work. No | politician, no matter how amateur, would arrange to change the clocks | merely because of Independence Day.
I think that's a stretch. It is certainly one thing that can be changed and would indicate that "things are different now" before any of the harder and more costly changes could be implemented. Whether that happened or not I don't know - but I certainly don't simply write it off as absurd.
| It'd be a major political blunder if | fiddling with the clocks is signaled as one of the most important things | the new legislature or executive could do.
I have seen no evidence of a claim like that, I would not be surprised to learn there were other issues considered considerably more important.
| That particular data entry must be wrong,
Sorry, "must be" is not an argument that has any traction with me at all, no matter what the issue. Unless you can show it is impossible (which clearly can't be done here, as it obviously would have been possible, regardless of how likely it might be considered) then only evidence of what actually happened will do.
Clearly, unless the assertion is that Mali was never in the -01:00 zone, there had to be a change sometime, as it now isn't.
Even if it turns out that it was a guess, and is wrong, that is no worse than the guess in the Honolulu zone is it? Should that one be removed too?
| It differs from all other Zones in its combination of UT offsets and | timezone abbreviations.
Eliminating abbreviations is another task that's been undertaken with some abandon - Honolulu's could be changed to "-10" then it would (post 1970) be identical to Etc/GMT+10 and you'd have one less zone to manage.
Furthermore, both those changes would surely be more fair and equitable?
kre
On 9/29/21 12:06 PM, Robert Elz wrote:
| That particular data entry must be wrong,
Sorry, "must be" is not an argument that has any traction with me at all, no matter what the issue.
That's not a viable standard. By that standard, someone could invent some data saying that Bamako was 4 hours ahead of GMT in 1938 and then say "prove me wrong". We shouldn't include data in tzdb merely because we can't (or lack the time to) prove it wrong. Here, the only evidence we have is Shanks, and Shanks has been shown to be wrong so often that it's almost *negative* evidence. Shanks often invented data for older timestamps that are harder to verify. He did it over and over again. Shanks's pre-1970 data is terrible data and we should rely on it as little as possible. Since you insist on questioning my admittedly uninformed suspicion of that particular data entry, I did a bit of legwork for it. That data entry comes from the Shanks & Pottenger International Atlas, 6th edition (2003), page 257 (Mali, Time Table # 1, and column 1 of the main data), which says that Bamako observed -01 from 1934-02-26 00:00 to 1960-06-20 00:00. However, 1959 American Nautical Almanac (published in 1957 in London by Her Majesty's Stationery Office and in Washington by the US Naval Observatory), the "STANDARD TIMES (Corrected to December 1956)" table, List II, page 264, says that French Sudan (which at the time existed and contained Bamako) observed G.M.T. Neither of these are primary sources, of course. That being said, of the two sources, the American Nautical Almanac (though often flawed) is typically more reliable than Shanks. At least some of that Shanks data item is most likely wrong. This is just one data item, and from the information given above it's not clear how to fix it. Nor do we have time to fix it. Nor do I want to (or have the time to) go through a similar exercise for all the other data items for Bamako (also quite possibly wrong) or for all the other data in 'backward' (also a lot of it wrong). But that's OK since this is just low-quality pre-1970 data in 'backward', and nobody (I hope) relies on 'backward' for anything important.
Eliminating abbreviations is another task that's been undertaken with some abandon
I eliminated only abbreviations that I had earlier invented with abandon and so were unsourced. 'HST' is a widely accepted English-language time zone abbreviation, which is why I didn't eliminate it.
On Tue, 28 Sep 2021, Paul Eggert via tz wrote:
On 9/28/21 11:59, Michael H Deckers wrote:
America/New_York, for example, merges many regions that did not observe the same time before 1970.
Yes, but it merged only zones that _never_ had an entry in tzdb. What we are doing now is merging tome zones that have existed separately for years within tzdb.
There are many examples of those merges too. For example, Europe/Zurich currently covers Liechtenstein even though before 2013 there was a separate Zone for Europe/Vaduz.
And IMO, there should be a seperate one for Europe/Vaduz if any data is different.
And anyway, saying "we've always done it that way" would not be an adequate response to valid fairness concerns.
"Fairness" is also not "because we don't have accurate enough pre-1970 data for one region, no other region can have this data either". Which is what you were doing with the Oslo/Stockholm/Berlin and Brussels/Amsterdam merges. Fairness would be to allow pre-1970 mostly accurate data to available for *all* (ISO) regions. I understand that for some use cases, people might want the smallest possible binary size. Make a switch for *that* use case instead, with the default being what we already had pre 2021b, or potentially even undoing some of the other destructive changes that have been brought in since you took over as TZ Coordinator. cheers, Derick
On 29.09.21 12:05, Derick Rethans via tz wrote:
I understand that for some use cases, people might want the smallest possible binary size. Make a switch for*that* use case instead, with the default being what we already had pre 2021b, or potentially even undoing some of the other destructive changes that have been brought in since you took over as TZ Coordinator.
SOME use cases? MOST use cases. The TZDB has always been meant to be small to fit on the smallest devices, which by the way, used to be rather large devices like Sun 3/50s. Now those small devices might be watches, lawn mowers, and other devices that FAR and away exceed the PHP user base. Eliot
On Wed, Sep 29, 2021 at 12:12:48PM +0200, Eliot Lear via tz wrote:
On 29.09.21 12:05, Derick Rethans via tz wrote:
I understand that for some use cases, people might want the smallest possible binary size. Make a switch for*that* use case instead, with the default being what we already had pre 2021b, or potentially even undoing some of the other destructive changes that have been brought in since you took over as TZ Coordinator.
SOME use cases? MOST use cases. The TZDB has always been meant to be small to fit on the smallest devices, which by the way, used to be rather large devices like Sun 3/50s. Now those small devices might be watches, lawn mowers, and other devices that FAR and away exceed the PHP user base.
If the intention is to be lean then the answer is to use the build flags to strip out all pre-1970 data. This provides a fairness advantage since no locales are more equal than any other. It is also easier to explain to users that nobody get data for pre-1970 than to explain why there are data for some completley unrelated place. /MF
participants (9)
-
Brian Inglis -
Derick Rethans -
Derick Rethans -
Eliot Lear -
Emily Crandall Fleischman -
Magnus Fromreide -
Michael H Deckers -
Paul Eggert -
Robert Elz