On 7/27/22 01:47, Stephen Colebourne via tz wrote:

Can you clarify what is generated?

With the patch, this shell command: make PACKRATDATA=backzone PACKRATLIST=zone.tab install installs into /usr/share/zoneinfo files that are identical to those installed by global-tz's "make install", except that /usr/share/zoneinfo/tzdata.zi's lines differ in ways that should not matter. To be precise, tzdata.zi's lines are sorted differently (which is irrelevant) and have one unused ruleset (also irrelevant, and easily fixed with the attached further patch which I have not installed either) and tzdata.zi's version comments differ as follows: -# version 2022agtz-122-gcd2e4b5 +# version 2022a-60-ge743ce9 +# ddeps backzone zone.tab None of this should affect how tzdata.zi is interpreted.

the tooling chains that global-tz supports need the *source* files (africa, europe, northamerica...) to contain the data

You can get that by using the tailored_tarballs target added recently. For example, this shell command: make DATAFORM=rearguard \ PACKRATDATA=backzone PACKRATLIST=zone.tab tailored_tarballs generates a tarball that should be suitable for the toolchains you mention. I just now did that using the two proposed patches, and put the results here temporarily: https://www.cs.ucla.edu/~eggert/tz/tzdata2022a-61-g3af508c-tailored.tar.gz

On Wed, 27 Jul 2022 at 19:10, Paul Eggert via tz <tz@iana.org> wrote:

On 7/27/22 01:47, Stephen Colebourne via tz wrote:

...

Can you clarify what is generated? the tooling chains that global-tz supports need the *source* files (africa, europe, northamerica...) to contain the data

You can get that by using the tailored_tarballs target added recently. For example, this shell command:

make DATAFORM=rearguard \ PACKRATDATA=backzone PACKRATLIST=zone.tab tailored_tarballs

generates a tarball that should be suitable for the toolchains you mention.

Thanks. The output being generated here is correct for some downstream toolchains AFAICT. Observations: 1) Some of the toolchains (ThreeTen-BP at least) depend on the `leapseconds` file, so can that be added please. 2) The generated tarball places all data in the etcetera file, rather than in africa/asia etc. This may cause problems for downstream users that wish to only package some files. I suspect that group of people is small, nevertheless it is worth noting that the generated tarball is not equivalent. 3) The generated tarball omits other files like zone.tab, iso3166.tab, backzone, backward and so on. In effect the contents are significantly less functional than those of global-tz. ie. global-tz goes to significant lengths to simulate exactly what iana-tz would have looked like if the pre-1970 merges had never happened. The output of global-tz is therefore fully compatible with all toolchains, no matter what they do with the data. Actions/Questions: As a minimum we need to add the `leapseconds` file to the generated tarball. Is it practical to keep the various files separate? Or to include more files? Right now it doesn't really feel like a compatible tzdata tarball as it has quite different content. My concern here is non-Java downstream projects that wish to avoid the merges - do they have what they need? (ie. I would prefer not to maintain global-tz, and it will need a good few hours work to undo the latest European merges, but I would need to be sure that all potential use cases are covered.) thanks Stephen

On 8/3/22 18:27, Paul Eggert wrote:

Including more files is easy and is done by the attached patch.

That patch was a little too enthusiastic for tailored tarballs in vanguard form which doesn't need 'pacificnew', so I installed the attached further patch. This patch doesn't affect tarballs in main or rearguard form, as they still have an empty pacificnew file for the benefit of TZUpdater 2.3.1 and earlier.

Date: Thu, 4 Aug 2022 22:54:45 +0100 From: Stephen Colebourne via tz <tz@iana.org> Message-ID: <CACzrW9A2PPmUz8A1=mkRCGwny-oA=Kjd=dTLjweE=X0+_-Umqw@mail.gmail.com> | I'd recommend anyone else that wishes to use a global-tz like | non-merged set of data to try the new tar.gz file to see if it meets | their needs. I have, while preparing to incorporate the update into NetBSD, and it doesn't seem to (unless I am missing something) do what we need at all. Generating the appropriate TZIf files is only half the battle - NetBSD users mostly have the sources (for everything) (as contracted with Linux, etc, where even though the sources are available, a very small fraction of users seem to make any use of that). What I need is for the data to be in the files it has always been in. Whether it gets there by simply unpacking the tzdata tar file, or by running some script upon what is there to produce the desired results is not important - but running a script to produce some other file, which isn't one where the user would expect to find data should they wish to change it, isn't useful to us at all. I would have also thought that keeping all of the relevant data in the appropriate files would make maintenance easier - when changes are being made, all the (likely) zones that might be affected are in one place - if you want to ignore some of them and make them links (essentially replacing data which has been characterised as possibly, or even probably, incorrect, with data which is almost guaranteed incorrect) then doing that as the data is processed seems like the better choice, rather than having to go check some other file, and make sure that everywhere that links have been made are still appropriate to remain links seems like the hard way to me. So, if you don't make a 2022btgz version available, I guess I will go back to manually inserting the updates, as I have done previously. kre eg: since we haven't yet released a NetBSD update for 2022b, any users in Chile who want to be prepared can easily make the update themselves, but they need to get the same results when they do "zic southamerica" (other than the change they made) as the TZif files that are distributed with the system. (And the same for europe, australasia, northamerica, africa, ...)

Date: Mon, 15 Aug 2022 13:24:48 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <fb33ecdc-753d-ea7e-6d4d-f30af7b0e059@cs.ucla.edu> | In what sense is that not true for the tarball format in question[1]? The data isn't in the files it belongs in. | What programs go awry if that tarball's data are used, and what are the | symptoms of these failures? One cannot easily checked what has changed, as the whole data file (all of them in that format) has gone away. I know you understand version control systems - we use one as well (currently cvs, gradually migrating to mercurial) and being able to look and see what changed in what version (and why) is important. | If the main problem is that a user who wants to hand-edit this data will | not know to edit 'etcetera' rather than (say) 'australasia', then it'd | be easy to change the tarball generator so that instead of 'australasia' | being empty it would contain a comment like "# Please edit the file | 'etcetera' instead."; would that help? No. If you would put the data which belongs in australasia back in the australasia file, that would help (and the same with all the others). Even if the file had markers in it, showing which parts belong to (or originate from if you prefer) which original file it would be better. That is, assuming that when the backzone data is merged in, it is put into the correct places in the file (that from which it was removed) so that we don't have countless differences which are in reality simply lines moving around. | [1] Here's the latest iteration of the tarball format in question: | https://www.cs.ucla.edu/~eggert/tz/tzdata2022b-5-gc9ba895-tailored.tar.gz Thanks -- in later reading of previously unread messages, I saw there was one of those for 2022a as well, had I seen that one I might have looked at it earlier, not that I think that would have changed anything. This file (as it doesn't include anything from backzone) doesn't help at all though - the tzdata file as distributed is easier to deal with than that particular one - but I assume that much is able to be altered. kre

On 8/16/22 00:08, Bradley White wrote:

Is the intent that the standard install and the "PACKRATDATA=backzone PACKRATLIST=zone.tab" install should not exhibit any differences for 1970+ timestamps?

Yes, and you can test this by running something like the following in a source directory. The diff command should output nothing. (The test takes a while.) I did this sort of test before releasing 2022b. make clean make TOPDIR=$PWD/tza ZFLAGS=-r@0 install ./zdump -i $(awk '/^[^#]/{print $3}' zone1970.tab) >a.tzs make clean make TOPDIR=$PWD/tzb ZFLAGS=-r@0 \ PACKRATDATA=backzone PACKRATLIST=zone.tab \ install ./zdump -i $(awk '/^[^#]/{print $3}' zone1970.tab) >b.tzs diff a.tzs b.tzs rm -r tza tzb a.tzs b.tzs

On 8/16/22 14:57, Bradley White wrote:

$ TZ=<path-to-standard>/Africa/Freetown date -d @1660625107 Tue Aug 16 04:45:07 GMT 2022 $ TZ=<path-to-packrat>/Africa/Freetown date -d @1660625107 Tue Aug 16 05:05:07 +01 2022

and ...

$ TZ=<path-to-standard>/Pacific/Saipan date -d @230659200 Sun Apr 24 03:00:00 GDT 1977 $ TZ=<path-to-packrat>/Pacific/Saipan date -d @230659200 Sun Apr 24 02:00:00 +10 1977

Thanks, good catches. The Africa/Freetown glitch was introduced in 2022a when fixing other problems with Sierra Leone. The Pacific/Saipan glitch is more interesting. It was introduced in 2018h, when we added DST rules for Pacific/Guam but I didn't notice that this should have caused Pacific/Saipan to split off and so I mistakenly kept the latter linked to the former. However, I just now looked into this and discovered that the old Pacific/Saipan data entries (taken from Shanks) were also mistaken. In the default build, as luck would have it the 2015h merge of Pacific/Guam and Pacific/Saipan, together with my 2018h mistake, actually *fixed* bugs in Pacific/Saipan because the merge and my subsequent mistake caused Shanks's mistakes to be ignored. I ran the following test to find similar glitches: rm -fr tza tzb make clean make TOPDIR=$PWD/tza ZFLAGS=-r@0 install ./zdump -i $(awk '/^[^#]/{print $3}' zone.tab) >a.tzs make clean make TOPDIR=$PWD/tzb ZFLAGS=-r@0 \ PACKRATDATA=backzone PACKRATLIST=zone.tab \ install ./zdump -i $(awk '/^[^#]/{print $3}' zone.tab) >b.tzs diff a.tzs b.tzs rm -r tza tzb a.tzs b.tzs and I found one more glitch: the time zone abbreviation for Pacific/Midway disagreed (this glitch was introduced in 2015b). I installed into the development repository the attached patch to 'backzone' to fix these glitches. The above test passes now. Because this patch affects only 'backzone', it doesn't affect any timestamps in the default build.

On 8/17/22 10:31, Bradley White wrote:

For what it's worth, that patch appears to remove the last instance of "DST all year" <https://github.com/eggert/tz/blob/aca1a705795d4d3986a3ac795e673c410ace2693/z...>. Hopefully you have some other hidden tests that continue to exercise those code paths.

Unfortunately all the tests I normally run are in the Makefile, and none of them exercise "DST all year" as far as I know. It would be nice if someone had the time to write good test cases for "DST all year". For example, POSIX requires support for it by via tricky settings like TZ='XXX6PDT7,0/0,J365/23' and tzcode should work with such settings (which I think it does). That being said, I expect that a lot of other software would misbehave under "DST all year". This is partly why tzdata has routinely modeled "DST all year" as standard time in the past. This may sound like a small thing, but it could turn into a real problem if the US adopts a "DST all year" law.

On 8/17/22 20:30, Bradley White wrote:

The sense I'm getting from discussions on the backzone issue is that many corporations and redistributors will become "PACKRAT"s when moving to 2022b, if only to provide historical consistency

I'm not getting the same sense. And if "historical consistency" means "don't mess with TZDB data", then redistributors should avoid the PACKRAT* options, as switching from 2022a to 2022b+PACKRATDATA introduces more changes to timestamps than simply upgrading from 2022a to 2022b. There are other good reasons not to use the PACKRAT* options. Not only do they add politics (which in the long run will likely cause trouble for people opting for PACKRAT*), they use lower-quality data that, despite repeated requests, nobody has volunteered to maintain. If you want stability this is not a good basis for it.

Concretely, I suggest that backzone changes affecting current timestamps (like the Africa/Freetown case above) be treated with the same release expediency that would be afforded to similar changes in the "default" data.

I don't see a need to issue a new TZDB release merely because of glitches involving the new options. Those who are competent and brave (or foolhardy :-) enough to use these bleeding-edge options can easily apply the patch published in <https://mm.icann.org/pipermail/tz/2022-August/031836.html>, assuming they care about the glitches. The rest of us can wait for the next release. As usual, though, my bottom-line advice is: don't use PACKRAT*.

Date: Mon, 15 Aug 2022 21:36:52 +0100 From: Stephen Colebourne via tz <tz@iana.org> Message-ID: <CACzrW9CP9w5SMUX=Er7ana2dyGn4oW0T18HB+mG6-Hg_kwdy_w@mail.gmail.com> I am going to try this off list, but it looks as if joda.org mail goes to gmail, in which case it is 99% certain that the message will never make it to you (Google don't like my attitudes wrt e-mail and the takeover by the major e-mail providers - they bounce almost all mail from munnari.oz.au). As I expected, that failed - something called mx.kundenserver.de received the message, then tried forwarding to gmail, which rejected the message. So apologies to people on the list who didn't really need to see this. | Robert, if there is something specific about global-tz that is "wrong" I would have preferred if the comments from backzone moved back to the original files, along with the data - the way it was done for 2022agtz but it worked as it was, in general. kre

On 4 August 2022 01:01:28 BST, Stephen Colebourne via tz <tz@iana.org> wrote:

3) The generated tarball omits other files like zone.tab, iso3166.tab, backzone, backward and so on.

My tool chain relies on the *.tab files at least. cheers Derick