Adding a "test" region?
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream. It's probably a good idea to have this sort of thing in place *before* some region makes a change that actually violates one of these assumptions. A few example test cases I can think of (but I assume others will have *many* other ideas): - Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year - Zone which goes from STD->DST with no change in offset. - Zone with new rules that don't take place until after 2038 - Zone with arbitrary DST offsets (10m15s, for example) - Zone that corresponds to TAI (e.g. offsets change to compensate for leap seconds) Downstream consumers could then use this in their test suites and be effectively "on notice" that they should be able to handle these sorts of cases, even though we have no examples of them existing in the wild (though there's actually some case to be made for interpreting TAI as a time zone, in which case that one *does* exist in the wild). There is some precedent for this sort of thing in the GREASE mechanism for testing TLS extensibility ( https://tools.ietf.org/html/draft-ietf-tls-grease-00 ), which provides some mechanism for testing whether TLS implementations will break if some arbitrary extensions are added. Best, Paul
On Jan 24, 2018, at 4:36 PM, Paul G <paul@ganssle.io> wrote:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream. It's probably a good idea to have this sort of thing in place *before* some region makes a change that actually violates one of these assumptions.
A few example test cases I can think of (but I assume others will have *many* other ideas):
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year - Zone which goes from STD->DST with no change in offset. - Zone with new rules that don't take place until after 2038 - Zone with arbitrary DST offsets (10m15s, for example) - Zone that corresponds to TAI (e.g. offsets change to compensate for leap seconds)
Downstream consumers could then use this in their test suites and be effectively "on notice" that they should be able to handle these sorts of cases, even though we have no examples of them existing in the wild (though there's actually some case to be made for interpreting TAI as a time zone, in which case that one *does* exist in the wild).
There is some precedent for this sort of thing in the GREASE mechanism for testing TLS extensibility ( https://tools.ietf.org/html/draft-ietf-tls-grease-00 ), which provides some mechanism for testing whether TLS implementations will break if some arbitrary extensions are added.
To be effective it would need to come with something like this: https://nodatime.github.io/tzvalidate/ Howard
Paul G said:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream.
That sounds a really good idea.
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year
Zone which changes every month, or even every week. -- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not. Of course it would be possible for the tools that track changes to ignore the test entry, but that's an incompatible change. And that also only works if the test zone(s) have rigorously defined name patterns fixed for all time. paul
On Jan 24, 2018, at 4:40 PM, Clive D.W. Feather <clive@davros.org> wrote:
Paul G said:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream.
That sounds a really good idea.
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year
Zone which changes every month, or even every week.
-- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
On 01/24/2018 04:55 PM, Paul.Koning@dell.com wrote:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker. That said, one of the problems this would be designed to solve is that apparently a lot of people are already *not* testing against the master branch or doing their own tests, since we hear about a lot of problems only *after* new releases come out. Having the default be "you don't get the test zones" would at least partially defeat the purpose of such an initiative.
Of course it would be possible for the tools that track changes to ignore the test entry, but that's an incompatible change. And that also only works if the test zone(s) have rigorously defined name patterns fixed for all time.
I assume it's fine to have rigorously defined name patterns. I was thinking of something designed from the beginning to be filtered out (it could, for example, *only* generate zones in TestZones/, and/or there can be a "testzones" file that lists all the fictional zones). Since it seems like Paul is already leaning towards moving to a 2-track release ("main" release and "compatibility" release that is more stable with respect to some of these features), it would make sense to distribute the test zones *only* as part of the "main" release (and whether the `TestZone/` region/folder is generated at all can be opt-in in `zic`, since this is mostly intended for downstream *compilers*, not for the much more stable compiled binaries). Presumably anyone not ready to handle the test zones can opt-out easily enough by using the compatibility releases. Best, Paul
On Jan 24, 2018, at 4:40 PM, Clive D.W. Feather <clive@davros.org> wrote:
Paul G said:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream.
That sounds a really good idea.
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year
Zone which changes every month, or even every week.
-- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
On Jan 24, 2018, at 5:24 PM, Paul G <paul@ganssle.io> wrote:
On 01/24/2018 04:55 PM, Paul.Koning@dell.com wrote:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker.
Sure. The storage products I work on (Dell EqualLogic storage arrays) allow the customer to set the array time zone, for friendly timestamps. The GUI offers the timezones named in the tzdata database. In the development organization, we have a script that downloads the current tzdata and updates the product source files to reflect the current information. Since it is unacceptable for geek internals like a test timezone to show up in a customer GUI, we would have to change our tools to ignore those entries, if tzdata started to include them. For that to happen there would have to be adequate announcement, and a commitment that there is a well defined name pattern that will always be used. Since there is no "Test" continent, naming them Test/<something> would clearly work. But even so, that is a change from the current tzdata content, which as defined today contains only meaningful real world data. paul
To be clear, this is mostly about testing tzdata compilers, not consumers of the compiled files (though it would also be useful for that). I think that this would come with a change to `zic` to add either a --test or a --no-test flag which would compile or not compile the test zones as appropriate. I'm weakly in favor of making the test region opt out (so by default running `zic` would generate them). No one further downstream than whoever distributes your zoneinfo binaries should bother with filtering out these zones, and for the zoneinfo distributors it would be a matter of adding a flag to the zic command. On January 24, 2018 10:52:58 PM UTC, Paul.Koning@dell.com wrote:
On Jan 24, 2018, at 5:24 PM, Paul G <paul@ganssle.io> wrote:
On 01/24/2018 04:55 PM, Paul.Koning@dell.com wrote:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker.
Sure. The storage products I work on (Dell EqualLogic storage arrays) allow the customer to set the array time zone, for friendly timestamps. The GUI offers the timezones named in the tzdata database. In the development organization, we have a script that downloads the current tzdata and updates the product source files to reflect the current information.
Since it is unacceptable for geek internals like a test timezone to show up in a customer GUI, we would have to change our tools to ignore those entries, if tzdata started to include them. For that to happen there would have to be adequate announcement, and a commitment that there is a well defined name pattern that will always be used. Since there is no "Test" continent, naming them Test/<something> would clearly work. But even so, that is a change from the current tzdata content, which as defined today contains only meaningful real world data.
paul
On Wed, Jan 24, 2018 at 6:14 PM, Paul G <paul@ganssle.io> wrote:
To be clear, this is mostly about testing tzdata compilers, not consumers of the compiled files (though it would also be useful for that). I think that this would come with a change to `zic` to add either a --test or a --no-test flag which would compile or not compile the test zones as appropriate.
I'm a consumer, and would love to have a "Test" area with stable, define edge cases in the default database so we could use these in our automated testing, rather than picking ones like Pacific/Tongatapu and Pacific/Kwajalein. But, we already filter the list so normal user's don't see Etc and similar, but power users can still use them if they need to.
Paul.Koning@dell.com said:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker.
Sure. The storage products I work on (Dell EqualLogic storage arrays) allow the customer to set the array time zone, for friendly timestamps. The GUI offers the timezones named in the tzdata database. In the development organization, we have a script that downloads the current tzdata and updates the product source files to reflect the current information.
Since it is unacceptable for geek internals like a test timezone to show up in a customer GUI,
(1) This is covered by the other Paul's comment about "erroneously display them to users". (2) Do you currently display the TAI zones, the GMT+n non-geographical zones, the backwards zones, etc.? I would have thought that was more confusing that test zones (particularly if someone confuses Europe/London with Etc/GMT). -- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
(2) Do you currently display the TAI zones, the GMT+n non-geographical zones, the backwards zones, etc.? I would have thought that was more confusing that test zones (particularly if someone confuses Europe/London with Etc/GMT).
And, I've just remembered, do you explain to your users that Etc/GMT-1 is for the zone GMT+1? -- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
On Jan 25, 2018, at 10:41 AM, Clive D.W. Feather <clive@davros.org> wrote:
Paul.Koning@dell.com said:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker.
Sure. The storage products I work on (Dell EqualLogic storage arrays) allow the customer to set the array time zone, for friendly timestamps. The GUI offers the timezones named in the tzdata database. In the development organization, we have a script that downloads the current tzdata and updates the product source files to reflect the current information.
Since it is unacceptable for geek internals like a test timezone to show up in a customer GUI,
(1) This is covered by the other Paul's comment about "erroneously display them to users".
(2) Do you currently display the TAI zones, the GMT+n non-geographical zones, the backwards zones, etc.? I would have thought that was more confusing that test zones (particularly if someone confuses Europe/London with Etc/GMT).
No, we don't, because we filter those. Obviously we can filter this new thing also. What concerns me is that it seems to have been suggested as something that could just be added without any backward compatibility concerns, and that is not correct. If people are given adequate notice of this new thing so they can update the tools in advance, and the test mechanism is standardized so that it can be reliably filtered, then it is probably ok. One remaining concern: the idea that changes to the Test data could be made periodically to exercise the machinery. That is ok if it coincides with real changes. But if this periodic thing causes tzdata releases to occur that have no real content change, then I object. paul
On 25 January 2018 at 11:20, <Paul.Koning@dell.com> wrote:
What concerns me is that it seems to have been suggested as something that could just be added without any backward compatibility concerns, and that is not correct. If people are given adequate notice of this new thing so they can update the tools in advance, and the test mechanism is standardized so that it can be reliably filtered, then it is probably ok.
If we are so concerned about introducing any potentially-breaking changes as it seems this list is, then this may even warrant an initial out-of-band email to tz-announce@. (And I think this is a big enough procedural change to warrant that anyway.) I'd imagine a procedure something like this: - Once there is sufficient consensus here on adding a Test namespace, we announce that the first release after a certain appointed date (e.g., ~90 days from announcement?) will have a new Test namespace. - Until the appointed date, business as usual for real data changes only. - On or after the appointed date, the Test namespace is created and added to the development branch, and the build process ignores it by default, but allows a flag to build it. It contains a single zone on perpetual -00. - The first regular release after that (again, based on real data changes) will contain this single-zone test file, and will also announce a date when actual test cases will be added. - We follow a similar procedure for adding the initial test cases, which reflect as accurately/succinctly as possible the actual limitations (and lack thereof) of the tzdb format as it exists today. - After a year or two, we issue a separate announcement that the first release after a certain appointed date will enable building the tests by default, but there will be a flag to ignore it, and we follow through as above. Principles we'd follow would include: - We never roll a new release just for test cases without the need to update real data, similar to the current *de facto* policy for code updates. The only exception might be if we ever manage to get to the point where we're going more than ~2–3 years between data updates (ha!), it may be worthwhile to continue the habit of preparing a release with tzcode and tests only and encouraging downstream updates, just so it isn't forgotten that despite the then-current stability, these things still aren't set in stone. - Over time, new "features" are added first as test cases (possibly even in an explicit Test/Beta sub-namespace) where development is deemed necessary — such as other calendars (which has already come-up and been worked around in the short term) or other timescales/epochs (TAI, Mars time??), which may otherwise crop up at any time. - Likewise, if future endeavors may require dropping or fiddling with certain aspects of POSIX-compliance, we test them first. This project has always and will necessarily continue to evolve, so let's take some steps in the short-term to ensure that we and our downstream consumers are ready to achieve those longer-term necessities. -- Tim Parenti
On Jan 25, 2018, at 1:06 PM, Tim Parenti <tim@timtimeonline.com<mailto:tim@timtimeonline.com>> wrote: On 25 January 2018 at 11:20, <Paul.Koning@dell.com<mailto:Paul.Koning@dell.com>> wrote: What concerns me is that it seems to have been suggested as something that could just be added without any backward compatibility concerns, and that is not correct. If people are given adequate notice of this new thing so they can update the tools in advance, and the test mechanism is standardized so that it can be reliably filtered, then it is probably ok. If we are so concerned about introducing any potentially-breaking changes as it seems this list is, then this may even warrant an initial out-of-band email to tz-announce@. (And I think this is a big enough procedural change to warrant that anyway.) I'd imagine a procedure something like this: * Once there is sufficient consensus here on adding a Test namespace, we announce that the first release after a certain appointed date (e.g., ~90 days from announcement?) will have a new Test namespace. * Until the appointed date, business as usual for real data changes only. * On or after the appointed date, the Test namespace is created and added to the development branch, and the build process ignores it by default, but allows a flag to build it. It contains a single zone on perpetual -00. * The first regular release after that (again, based on real data changes) will contain this single-zone test file, and will also announce a date when actual test cases will be added. * We follow a similar procedure for adding the initial test cases, which reflect as accurately/succinctly as possible the actual limitations (and lack thereof) of the tzdb format as it exists today. * After a year or two, we issue a separate announcement that the first release after a certain appointed date will enable building the tests by default, but there will be a flag to ignore it, and we follow through as above. Principles we'd follow would include: * We never roll a new release just for test cases without the need to update real data, similar to the current de facto policy for code updates. The only exception might be if we ever manage to get to the point where we're going more than ~2–3 years between data updates (ha!), it may be worthwhile to continue the habit of preparing a release with tzcode and tests only and encouraging downstream updates, just so it isn't forgotten that despite the then-current stability, these things still aren't set in stone. * Over time, new "features" are added first as test cases (possibly even in an explicit Test/Beta sub-namespace) where development is deemed necessary — such as other calendars (which has already come-up and been worked around in the short term) or other timescales/epochs (TAI, Mars time??), which may otherwise crop up at any time. * Likewise, if future endeavors may require dropping or fiddling with certain aspects of POSIX-compliance, we test them first. This project has always and will necessarily continue to evolve, so let's take some steps in the short-term to ensure that we and our downstream consumers are ready to achieve those longer-term necessities. -- Tim Parenti Thanks Tim. That all sounds good to me. I'm comfortable with proceeding in the way you described. paul
On 2018-01-24 15:24, Paul G wrote:
On 01/24/2018 04:55 PM, Paul.Koning@dell.com wrote:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker.
See Hawai'i Missile Alert technical articles or search Risks list for "test in production" or "test data in production database" articles. How much do you want to further delay downstream distribution of changes? As someone else mentioned about Debian, the current release on most major distributions is 2017c, often on ...-update or even ...-security repos to get changes distributed! Package maintainers and QA have enough work to do to get stable releases thru their processes without introducing deliberate errors that get a release put on the back burner until the *volunteers* have time to deal with broken packages, or release a package that breaks downstream applications before it gets detected, complained about, and reverted. Distros are already pretty selective about what they package from a release, as looking at the selection of files available in a few distros' tzdata binary and source packages will show. The non-standard (non-existent) structure of the release probably requires package scripts to at least move each file into a directory structure acceptable to the distro build and packaging tools, add build and packaging control files, and/or patch distributed files to conform to their tools and processes, then get the builds and packages to a state where they can pass their QA processes to become officially released binary and source packages. As a result, additions to a release are likely to be ignored until that breaks a process or fails QA e.g. moving version from Makefile to a separate file. QA processes are likely to involve regression testing all downstream packages dependent on tzdata, so any change that causes a downstream package issue can halt tzdata packaging until the issue can be reported, investigated, diagnosed, fixed, tested, and either resolved or the test bypassed for minor issues. Unfortunately, not all (no?) libraries use the tzcode reference implementation, as they also have to support i18n and l14n facilities, so QA will presumably run libc, coreutils, and other package tests across multiple locales, increasing the possibility of regressions.
That said, one of the problems this would be designed to solve is that apparently a lot of people are already *not* testing against the master branch or doing their own tests, since we hear about a lot of problems only *after* new releases come out. Having the default be "you don't get the test zones" would at least partially defeat the purpose of such an initiative.
As some major distributors have previously mentioned, their orgs have policies that they can only use official releases from official sources, and do not have either the authorization or resources to test against the experimental github releases. The result is that most significant complaints are made by major distributors only after official releases. Some users perform their own QA against the experimental github repo and they contribute reports and patches that will avoid some problems when an official release is generated.
Of course it would be possible for the tools that track changes to ignore the test entry, but that's an incompatible change. And that also only works if the test zone(s) have rigorously defined name patterns fixed for all time.
I assume it's fine to have rigorously defined name patterns. I was thinking of something designed from the beginning to be filtered out (it could, for example, *only* generate zones in TestZones/, and/or there can be a "testzones" file that lists all the fictional zones).
Since it seems like Paul is already leaning towards moving to a 2-track release ("main" release and "compatibility" release that is more stable with respect to some of these features), it would make sense to distribute the test zones *only* as part of the "main" release (and whether the `TestZone/` region/folder is generated at all can be opt-in in `zic`, since this is mostly intended for downstream *compilers*, not for the much more stable compiled binaries). Presumably anyone not ready to handle the test zones can opt-out easily enough by using the compatibility releases.
The official release must then remain the main stable compatibility release for consumption by the major downstream distributors and users. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
From: Brian Inglis <Brian.Inglis@SystematicSw.ab.ca> Date: Thu, 25 Jan 2018 22:45:53 -0700 Subject: Re: [tz] Adding a "test" region? | As someone else mentioned about Debian, the current release on most major | distributions is 2017c, What's the point there supposed to be? Until earlier this week 2017c was the last announced released version - until they upgrade to what was announced earlier this week, that is what they should be using? (2017 was a truly remarkable year, just 3 tz versions needed!) It is hard to imagine that you're faulting them for taking more than 2 days to get the upgrade done, nor that you're attempting to point out that it takes more than 2 days after tzdata is released until it is available to users. So, what was the point? | As some major distributors have previously mentioned, their orgs | have policies that they can only use official releases from official | sources, If "use" there means "release" that makes a lot of sense. But ... | and do not have either the authorization or resources to test against | the experimental github releases. Any organisation which claims to be a "major distributor" which doesn't have the resources to perform tests against an upcoming release, or even worse, prohibits that for some policy reason, should be ignored by all of us. That's unconscienable (and I find it hard to believe that any such thing actually exists.) Ignoring tzdata for a minute, do you really believe that the vendors of software packages that run on linux, or the orgs that package linux distros aren't testing the latest experimental linux kernels, to make sure that nothing breaks? | The result is that most significant complaints are made by major | distributors only after official releases. Which makes those complaints (for them) almost useless. We get the benefit, as when we receive the complaint (if it is a real problem and not just some misunderstanding, or end user mistake) we get to fix the problem. But the distributor gets no benefit of that, as the last official release still has the bug/problem/issue in it (which may mean tat they cannot use it - and if they'er only allowed to use official releases, they can't patch locally either). Of course, "we" will eventually issue another release which will have that issue fixed - but it is also going to have everyone else's issues fixed, and most likely, other changes as well. It is entirely possible that the "major distributor" will find another issue with this release, which they can complain about, and we can fix, but now they're running 2 releases behind, and their users start to complain - have this happen 2 or 3 more times, and their release is so ancient that no-one will trust it any more. No sane distributor could permit such a possibility, let alone enforce it. Testing against known forthcoming releases (or everything they consume), ahead of them becoming "official releases" should be mandatory for their development/integration staff. Insane distributors, attempting to save a few pennies by assuming that someone else will find all the issues they would have found had they bothered to allocate the resources for pre-release testing (of other people's releases) deserve whatever problems they encounter after the release is issued. We should have no sympathy for them at all. Nor should we adjust (in any way) our release policy, or the contents, of the releases, and nor should we be issuing "emergency reversion" releases to cater for their lack of foresight. kre
On 01/26/2018 12:45 AM, Brian Inglis wrote:
The official release must then remain the main stable compatibility release for consumption by the major downstream distributors and users.
I *very strenuously* disagree with this. I think in general the default should be the latest version of the database, and any compatibility release should be something you have to opt in to. Generally people are not going to pick anything except the main release, and they are not going to make any changes unless that breaks. The whole point of a second release is that if some change (possibly necessary) breaks enough downstream consumers, it might be worth providing a release without these changes so that they don't have to manually patch all new releases themselves. People who are not broken will not seek out the alternate source because nothing breaks for them - and people writing new software will likely start to rely on the same undocumented features that the old, broken software uses. That said, I am also not entirely convinced that a second "compatibility" release is warranted. Even the Ireland case did not break backwards compatibility in any meaningful way - as has been said many times, zic has had the capability to handle negative DST offsets for decades, this is just the first example of the actual *use* of this feature. I think the main reason this is controversial is that this is, in some sense, an "opt in" change, because we can choose to define the "standard" offset as the one that allows all other offsets in the zone to be defined with positive SAVE values. In most other cases where major software has made inappropriate assumptions about the nature of the data, we will not get to "opt out", and if a "compatibility" release is provided, it will necessarily contain unambiguously inaccurate data. Rather than maintain a release that is the lowest common denominator of all the inaccuracies that are required to allow all the broken software out there to work, I think it should probably be incumbent upon people deploying the broken software to add in these inaccuracies to the data *a la carte* to the data before consuming it, so that they can have the *most* accurate data that their software can handle. Best, Paul
On 2018-01-24 15:24, Paul G wrote:
On 01/24/2018 04:55 PM, Paul.Koning@dell.com wrote:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Do you have examples of this? I'm not really sure what problem it would cause to have well-namespaced fictional zones in the real database. What kind of automated processes would be affected by there being some zones that don't represent real zones? I could imagine there being some applications that don't know about these test zones that erroneously display them to users, but why would users set their time zone to `TestZone/Test_Monthly_DST` or something? Doesn't seem hugely risky to me, but I would not be surprised to find that there are some applications where this could be a deal-breaker.
See Hawai'i Missile Alert technical articles or search Risks list for "test in production" or "test data in production database" articles.
How much do you want to further delay downstream distribution of changes? As someone else mentioned about Debian, the current release on most major distributions is 2017c, often on ...-update or even ...-security repos to get changes distributed! Package maintainers and QA have enough work to do to get stable releases thru their processes without introducing deliberate errors that get a release put on the back burner until the *volunteers* have time to deal with broken packages, or release a package that breaks downstream applications before it gets detected, complained about, and reverted.
Distros are already pretty selective about what they package from a release, as looking at the selection of files available in a few distros' tzdata binary and source packages will show. The non-standard (non-existent) structure of the release probably requires package scripts to at least move each file into a directory structure acceptable to the distro build and packaging tools, add build and packaging control files, and/or patch distributed files to conform to their tools and processes, then get the builds and packages to a state where they can pass their QA processes to become officially released binary and source packages. As a result, additions to a release are likely to be ignored until that breaks a process or fails QA e.g. moving version from Makefile to a separate file.
QA processes are likely to involve regression testing all downstream packages dependent on tzdata, so any change that causes a downstream package issue can halt tzdata packaging until the issue can be reported, investigated, diagnosed, fixed, tested, and either resolved or the test bypassed for minor issues. Unfortunately, not all (no?) libraries use the tzcode reference implementation, as they also have to support i18n and l14n facilities, so QA will presumably run libc, coreutils, and other package tests across multiple locales, increasing the possibility of regressions.
That said, one of the problems this would be designed to solve is that apparently a lot of people are already *not* testing against the master branch or doing their own tests, since we hear about a lot of problems only *after* new releases come out. Having the default be "you don't get the test zones" would at least partially defeat the purpose of such an initiative.
As some major distributors have previously mentioned, their orgs have policies that they can only use official releases from official sources, and do not have either the authorization or resources to test against the experimental github releases. The result is that most significant complaints are made by major distributors only after official releases. Some users perform their own QA against the experimental github repo and they contribute reports and patches that will avoid some problems when an official release is generated.
Of course it would be possible for the tools that track changes to ignore the test entry, but that's an incompatible change. And that also only works if the test zone(s) have rigorously defined name patterns fixed for all time.
I assume it's fine to have rigorously defined name patterns. I was thinking of something designed from the beginning to be filtered out (it could, for example, *only* generate zones in TestZones/, and/or there can be a "testzones" file that lists all the fictional zones).
Since it seems like Paul is already leaning towards moving to a 2-track release ("main" release and "compatibility" release that is more stable with respect to some of these features), it would make sense to distribute the test zones *only* as part of the "main" release (and whether the `TestZone/` region/folder is generated at all can be opt-in in `zic`, since this is mostly intended for downstream *compilers*, not for the much more stable compiled binaries). Presumably anyone not ready to handle the test zones can opt-out easily enough by using the compatibility releases.
On 2018-01-24 21:55:53 (+0000), Paul.Koning@dell.com wrote:
No, please do NOT do that. Anyone who depends on an automated or semi-automated system to track real changes will get completely messed up by such a tactic. Let the real database contain real data. A test database is fine; a test zone in the real data is not.
Of course it would be possible for the tools that track changes to ignore the test entry, but that's an incompatible change. And that also only works if the test zone(s) have rigorously defined name patterns fixed for all time.
I don't think we should push gratuitous changes to a Test/ (Utopia/, Dystopia/, Whatever/) region but we can easily get away with slipping it in with the next 'real' update to actually existing regions. Philip -- Philip Paeps Senior Reality Engineer Ministry of Information
On 24 January 2018 at 17:26, Philip Paeps <philip@trouble.is> wrote:
I don't think we should push gratuitous changes to a Test/ (Utopia/, Dystopia/, Whatever/) region but we can easily get away with slipping it in with the next 'real' update to actually existing regions.
Precisely. We wouldn't actually roll a release for changes to test data only. But tests could be pushed to the development repo at any time and should — ahem — *be tested.* And if people are directly parsing tzdata source files anyway, they already need to know which files they're looking at. If we added a file 'test' to express time zones on this fictional continent of 'test', and some downstream project somehow accidentally picked that up, that's due to their overbroad scope as it already is. If we really think downstream projects are too brittle for even *this* (but goodness, they shouldn't be)… we could, as a first step, just to have a single zone on perpetual -00 in a new "test" namespace and see what that breaks. Then after some reasonably brief time we could add actual test cases. -- Tim Parenti
Tim Parenti said:
If we really think downstream projects are too brittle for even *this* (but goodness, they shouldn't be)??? we could, as a first step, just to have a single zone on perpetual -00 in a new "test" namespace and see what that breaks. Then after some reasonably brief time we could add actual test cases.
I would support this. -- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
On 2018-01-24 16:36:45 (-0500), Paul G wrote:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream. It's probably a good idea to have this sort of thing in place *before* some region makes a change that actually violates one of these assumptions.
A few example test cases I can think of (but I assume others will have *many* other ideas):
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year - Zone which goes from STD->DST with no change in offset. - Zone with new rules that don't take place until after 2038 - Zone with arbitrary DST offsets (10m15s, for example) - Zone that corresponds to TAI (e.g. offsets change to compensate for leap seconds)
I think this is an excellent idea. But it should be a very obviously fictional region -- obvious enough even to end consumers that it can be distributed default enabled and not hidden behind a build-time knob nobody will ever touch. So it should not look like a test to the casual lazy distributor who doesn't want to distribute test code. Suggestions: Dystopia/Isle_of_Negative_Save Dystopia/Half_Hour_Bay Dystopia/Fifteen_Minutes_Late_Land Dystopia/Fifteen_Minutes_Early_Town Dystopia/Twice_Yearly_DST_Ville Dystopia/... It can be Utopia too. It can be anything other than "Test". People will just disable "Test" and not distribute it.
Downstream consumers could then use this in their test suites and be effectively "on notice" that they should be able to handle these sorts of cases, even though we have no examples of them existing in the wild (though there's actually some case to be made for interpreting TAI as a time zone, in which case that one *does* exist in the wild).
If it's going to be surrounded by "test" fences, nobody is every going to actually test it. If we make it obviously fictional, it can be on by default an distributed with the rest of the data. Philip -- Philip Paeps Senior Reality Engineer Ministry of Information
On 2018-01-24 23:22:19 (+0100), Philip Paeps wrote:
On 2018-01-24 16:36:45 (-0500), Paul G wrote:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream. It's probably a good idea to have this sort of thing in place *before* some region makes a change that actually violates one of these assumptions.
A few example test cases I can think of (but I assume others will have *many* other ideas):
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year - Zone which goes from STD->DST with no change in offset. - Zone with new rules that don't take place until after 2038 - Zone with arbitrary DST offsets (10m15s, for example) - Zone that corresponds to TAI (e.g. offsets change to compensate for leap seconds)
I think this is an excellent idea. But it should be a very obviously fictional region -- obvious enough even to end consumers that it can be distributed default enabled and not hidden behind a build-time knob nobody will ever touch. So it should not look like a test to the casual lazy distributor who doesn't want to distribute test code.
Suggestions:
Dystopia/Isle_of_Negative_Save Dystopia/Half_Hour_Bay Dystopia/Fifteen_Minutes_Late_Land Dystopia/Fifteen_Minutes_Early_Town Dystopia/Twice_Yearly_DST_Ville Dystopia/...
It can be Utopia too. It can be anything other than "Test". People will just disable "Test" and not distribute it.
Downstream consumers could then use this in their test suites and be effectively "on notice" that they should be able to handle these sorts of cases, even though we have no examples of them existing in the wild (though there's actually some case to be made for interpreting TAI as a time zone, in which case that one *does* exist in the wild).
If it's going to be surrounded by "test" fences, nobody is every going to actually test it. If we make it obviously fictional, it can be on by default an distributed with the rest of the data.
Come to think of it - it may be wiser to make it a subregion rather than a new region. Possibly under Arctic since there's only one zone there with a population of well under 10,000 humans. Arctic/Dystopia/... Rationale: systems may get confused if the top-level regions increase but are likely already more or less accustomed to changes at the second level. Philip -- Philip Paeps Senior Reality Engineer Ministry of Information
I really disagree with making it a sub-region of an arbitrary geographical region. That will just cause confusion as to whether it's real or not. If it's under "TestZones" it will be obvious enoguh. Plus, "number of regions never increases" is an unwarranted assumption, so that alone argues for adding it as a separate region. Of course this is going to break some software (or really, make it obvious that software is already broken) - the point is to break the software *now* when the maintainers of said broken software can just filter out the `TestZone` region while they are working on removing the unwarranted assumptions, rather than break it when Iceland decides (or rather, recognizes) that leap seconds are a bad idea and changes their official timekeeping to a TAI-based zone. On 01/24/2018 05:30 PM, Philip Paeps wrote:
On 2018-01-24 23:22:19 (+0100), Philip Paeps wrote:
On 2018-01-24 16:36:45 (-0500), Paul G wrote:
All this hubbub about the negative SAVE values in the tzdb source code makes it pretty obvious that downstream consumers are making some assumptions about undocumented features of the tzdb code. I'm wondering if it might be worth it to add a new region that contains fictional time zones that have properties that are perfectly valid but are edge cases that might be improperly handled downstream. It's probably a good idea to have this sort of thing in place *before* some region makes a change that actually violates one of these assumptions.
A few example test cases I can think of (but I assume others will have *many* other ideas):
- Zone with negative DST - Zone with large DST offset and/or large standard offset - Zone with 3, 4 or 5 offsets per year - Zone which goes from STD->DST with no change in offset. - Zone with new rules that don't take place until after 2038 - Zone with arbitrary DST offsets (10m15s, for example) - Zone that corresponds to TAI (e.g. offsets change to compensate for leap seconds)
I think this is an excellent idea. But it should be a very obviously fictional region -- obvious enough even to end consumers that it can be distributed default enabled and not hidden behind a build-time knob nobody will ever touch. So it should not look like a test to the casual lazy distributor who doesn't want to distribute test code.
Suggestions:
Dystopia/Isle_of_Negative_Save Dystopia/Half_Hour_Bay Dystopia/Fifteen_Minutes_Late_Land Dystopia/Fifteen_Minutes_Early_Town Dystopia/Twice_Yearly_DST_Ville Dystopia/...
It can be Utopia too. It can be anything other than "Test". People will just disable "Test" and not distribute it.
Downstream consumers could then use this in their test suites and be effectively "on notice" that they should be able to handle these sorts of cases, even though we have no examples of them existing in the wild (though there's actually some case to be made for interpreting TAI as a time zone, in which case that one *does* exist in the wild).
If it's going to be surrounded by "test" fences, nobody is every going to actually test it. If we make it obviously fictional, it can be on by default an distributed with the rest of the data.
Come to think of it - it may be wiser to make it a subregion rather than a new region. Possibly under Arctic since there's only one zone there with a population of well under 10,000 humans.
Arctic/Dystopia/...
Rationale: systems may get confused if the top-level regions increase but are likely already more or less accustomed to changes at the second level.
Philip
On 2018-01-24 17:37:04 (-0500), Paul G wrote:
I really disagree with making it a sub-region of an arbitrary geographical region. That will just cause confusion as to whether it's real or not. If it's under "TestZones" it will be obvious enoguh. Plus, "number of regions never increases" is an unwarranted assumption, so that alone argues for adding it as a separate region.
Yeah. The subzone probably isn't a great idea. Just having something obviously (to endusers) but not too obviously (to downstream consumers averse to testing) fictional at the top-level is probably the best way forward. Calling it "Test" is almost 100% guaranteed to ensure it'll never actually be tested. May as well hide it in the Makefile somewhere.
Of course this is going to break some software (or really, make it obvious that software is already broken) - the point is to break the software *now* when the maintainers of said broken software can just filter out the `TestZone` region while they are working on removing the unwarranted assumptions, rather than break it when Iceland decides (or rather, recognizes) that leap seconds are a bad idea and changes their official timekeeping to a TAI-based zone.
Agree. Though I feel it would be better if it were less obvious to filter. I'd prefer an end user spend two milliseconds wondering 'where is Negative Save Land? ... ooh! Fictional ocean. Cute easter egg!' to distributors not testing ever. Philip -- Philip Paeps Senior Reality Engineer Ministry of Information
Philip Paeps said:
Calling it "Test" is almost 100% guaranteed to ensure it'll never actually be tested. May as well hide it in the Makefile somewhere. [...] Though I feel it would be better if it were less obvious to filter. I'd prefer an end user spend two milliseconds wondering 'where is Negative Save Land? ... ooh! Fictional ocean. Cute easter egg!' to distributors not testing ever.
If a distributor or other user of TZ doesn't test the test data, then as far as I'm concerned they have absolutely no grounds to complain when real changes break their systems. -- Clive D.W. Feather | If you lie to the compiler, Email: clive@davros.org | it will get its revenge. Web: http://www.davros.org | - Henry Spencer Mobile: +44 7973 377646
On 25 January 2018 at 15:36, Clive D.W. Feather <clive@davros.org> wrote:
If a distributor or other user of TZ doesn't test the test data, then as far as I'm concerned they have absolutely no grounds to complain when real changes break their systems.
As I have outlined in the other thread, just because a parser can parse the TZDB source file, does not mean that there is no problem. The problems occur in the interactions between the many and various systems in the time stack. A test file plays a relatively small role in testing the stack. Stephen
From: Stephen Colebourne <scolebourne@joda.org> Date: Thu, 25 Jan 2018 16:11:41 +0000 Subject: Re: [tz] Adding a "test" region? | As I have outlined in the other thread, just because a parser can | parse the TZDB source file, does not mean that there is no problem. This is absolutely correct. | The problems occur in the interactions between the many and various | systems in the time stack. A test file plays a relatively small role | in testing the stack. Of course. But the expectation is that you'd also run all of the other tests for all of the other interfaces that you have (I am sure your team of paid professionals has created such tests) with the test time zone. That way you get to discover whether any of your interfaces have problems, before things are being deployed in real timezones, and have time to make corrections. kre
On 25 January 2018 at 16:44, Robert Elz <kre@munnari.oz.au> wrote:
Of course. But the expectation is that you'd also run all of the other tests for all of the other interfaces that you have (I am sure your team of paid professionals has created such tests) with the test time zone.
I'm not part of CLDR, nor am I paid to work on date/time issues.
That way you get to discover whether any of your interfaces have problems, before things are being deployed in real timezones, and have time to make corrections.
It can be tested for a specific stack, but not for all possible stacks (as I indicated in the other thread). As such, I think a test file might at best act as an early warning of what will be a problem, not necessarily as a way to get those problems solved. Stephen
From: Stephen Colebourne <scolebourne@joda.org> Date: Thu, 25 Jan 2018 16:53:22 +0000 Subject: Re: [tz] Adding a "test" region? | I'm not part of CLDR, nor am I paid to work on date/time issues. But, it was earlier said, someone apparently is, right? | It can be tested for a specific stack, but not for all possible stacks | (as I indicated in the other thread). Why? Obviously each stack developer only tests their own, it is not a case of one person or organisation being expected to test everything, that would obviously be absurd. But if there was a zone designed for testing (or a set of such zones) then each developer could fairly easily (I would hope) test their stack (or stacks if they are responsible for more than one) using that zone (in addition to the others the test suites presumably already use.) I would certainly augment the NetBSD time, timezone and date-parsing tests to include such a zone if one were available. | As such, I think a test file might at best act as an early warning | of what will be a problem, not necessarily as a way to get those | problems solved. Of course, that's the idea - they let you know what fails (if anything) before the failures are present in a zone that is important (if some stack fails on the test zone, that doesn't prevent it still being used for more normal zones, while whatever it is is yet to appear there). Tests like this are also helpful in making sure that code changes do not break things - and for that, you want tests that really push the corner cases. A test zone with wildly improbable (but possible) data in it would be extremely useful for such scenarios. And no, I don't care what it is called (just as long as someone tells me!) It could be Australia/New_York for all I care. kre
On Wed, Jan 24, 2018 at 11:30 PM, Philip Paeps <philip@trouble.is> wrote:
Come to think of it - it may be wiser to make it a subregion rather than a new region. Possibly under Arctic since there's only one zone there with a population of well under 10,000 humans.
Arctic/Dystopia/...
Rationale: systems may get confused if the top-level regions increase but are likely already more or less accustomed to changes at the second level.
Isnt' just the point of the proposal, to challenge bad assumptions?
On 2018-01-24 23:42:18 (+0100), Pierpaolo Bernardi wrote:
On Wed, Jan 24, 2018 at 11:30 PM, Philip Paeps <philip@trouble.is> wrote:
Come to think of it - it may be wiser to make it a subregion rather than a new region. Possibly under Arctic since there's only one zone there with a population of well under 10,000 humans.
Arctic/Dystopia/...
Rationale: systems may get confused if the top-level regions increase but are likely already more or less accustomed to changes at the second level.
Isnt' just the point of the proposal, to challenge bad assumptions?
Yeah. Nevermind the subzone brainfart. But I stand by something like "Utopia" over "Test". Consumers like Paul at Dell and others who are on this list and presumably actually test tzdata can easily filter Utopia if they like. More casual consumers who don't know or care about this list are more likely to "inadvertently test" something like Utopia rather than Test. And if it breaks for them, it'll be obvious to them. If it doesn't break (Great!), their consumers are unlikely to complain too loudly about something as harmless as a couple of fictional places in their timezone GUI. Philip -- Philip Paeps Senior Reality Engineer Ministry of Information
It'd make sense to add a source file 'test.zi' to contain test cases. (I'd like new zic input files to end in '.zi'.) Some thought would have to go into the test cases, though. For example, how far away from zero could the UT offsets be? Even tzdata has some limits, which are not documented nor (I suspect) even known.
On Jan 24, 2018, at 5:42 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
It'd make sense to add a source file 'test.zi' to contain test cases. (I'd like new zic input files to end in '.zi'.) Some thought would have to go into the test cases, though. For example, how far away from zero could the UT offsets be? Even tzdata has some limits, which are not documented nor (I suspect) even known.
A separate input file is the way to do this. test.zi (or test_zones.zi) sounds good to me. Compilers can then go into test mode with nothing more than choosing which database files to read. And we need to explicitly state what we expect this input file to produce. Perhaps a test_zones.zo containing: Version: 2018d Test/Zone1 Initially: -00:00:52 standard LMT 1918-01-01 00:00:52Z +00:00:00 standard GMT 1920-09-01 00:00:00Z +00:20:00 daylight +0020 1920-12-30 23:40:00Z +00:00:00 standard GMT … Howard
participants (12)
-
Brian Inglis -
Clive D.W. Feather -
Howard Hinnant -
Matthew Donadio -
Paul Eggert -
Paul G -
Paul.Koning＠dell.com -
Philip Paeps -
Pierpaolo Bernardi -
Robert Elz -
Stephen Colebourne -
Tim Parenti