make rearguard_tarballs fails on macOS
Hi, The change to Makefile for 2020c makes it fail on macOS, as the touch command on Darwin does not have a -d option:
: The dummy pacificnew pacifies TZUpdater 2.3.1 and earlier. touch -md 2020-10-12T22:53:00Z \ tzdata2020c-rearguard.dir/pacificnew touch: illegal option -- d usage: touch [-A [-][[hh]mm]SS] [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ... make[1]: *** [tzdata2020c-rearguard.tar.gz] Error 1 make: *** [rearguard_tarballs] Error 2
Debbie
On 10/16/20 7:41 PM, Deborah Goldsmith via tz wrote:
The change to Makefile for 2020c makes it fail on macOS, as the touch command on Darwin does not have a -d option:
Thanks for reporting that. I installed the attached proposed patch into the development repository on GitHub. macOS uses FreeBSD utilities, right? According to the FreeBSD man page server, FreeBSD 'touch' has supported -d since FreeBSD 10 (2014). Is macOS still using older FreeBSD 'touch', or is the problem limited to older macOS releases (and if so, which ones)? In the meantime, you should be able to work around the problem by building rearguard tarballs on GNU/Linux, or on any other platform conforming to POSIX-2008 or later that has a working 'tar' program.
Hi Paul, This is on the latest version of macOS. While Darwin has a lot of content from FreeBSD, there’s no effort to keep the two in sync AFAIK. According to Wikipedia Darwin is not POSIX-compatible but is compatible with the Single UNIX Specification, version 3. I’ll just patch the Makefile and then patch the output to remove the “dirty”. Thanks, Debbie
On Oct 16, 2020, at 8:02 PM, Paul Eggert <eggert@CS.UCLA.EDU> wrote:
On 10/16/20 7:41 PM, Deborah Goldsmith via tz wrote:
The change to Makefile for 2020c makes it fail on macOS, as the touch command on Darwin does not have a -d option:
Thanks for reporting that. I installed the attached proposed patch into the development repository on GitHub.
macOS uses FreeBSD utilities, right? According to the FreeBSD man page server, FreeBSD 'touch' has supported -d since FreeBSD 10 (2014). Is macOS still using older FreeBSD 'touch', or is the problem limited to older macOS releases (and if so, which ones)?
In the meantime, you should be able to work around the problem by building rearguard tarballs on GNU/Linux, or on any other platform conforming to POSIX-2008 or later that has a working 'tar' program. <0001-Port-make-rearguard_tarballs-to-macOS.patch>
On 10/16/20 9:09 PM, Deborah Goldsmith wrote:
According to Wikipedia Darwin is not POSIX-compatible but is compatible with the Single UNIX Specification, version 3.
The Single UNIX Specification version 3 is the same as POSIX-2001, so I guess 2001 is as far as Apple has wanted to go, at least so far. Wikipedia says AIX and HP-UX also don't go past POSIX-2001, so today's fix should be helpful on those old OSes too.
On 2020-10-16, at 22:09:19, Deborah Goldsmith via tz <tz@iana.org> wrote:
This is on the latest version of macOS. While Darwin has a lot of content from FreeBSD, there’s no effort to keep the two in sync AFAIK. According to Wikipedia Darwin is not POSIX-compatible but is compatible with the Single UNIX Specification, version 3.
It seems to be in current Single UNIX: -d date_time Use the specified date_time instead of the current time. The option-argument shall be a string of the form: YYYY-MM-DDThh:mm:SS[.frac][tz] or: YYYY-MM-DDThh:mm:SS[,frac][tz] ... -t time Use the specified time instead of the current time. The option-argument shall be a decimal number of the form: [[CC]YY]MMDDhhmm[.SS] Why do we need both? Existing Art, I suppose. RATIONALE ... • In System V, an ambiguity exists when a pathname that is a decimal number leads the operands; it is treated as a time value. In BSD, no time value is allowed; files may only be touched to the current time. The -t time construct solves these problems for future conforming applications (note that the -t option is not historical practice).
I’ll just patch the Makefile and then patch the output to remove the “dirty”.
-- gil
Date: Fri, 16 Oct 2020 20:02:41 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <c49c9641-9e74-a378-e3d2-62826072a403@cs.ucla.edu> | macOS uses FreeBSD utilities, right? According to the FreeBSD man page | server, FreeBSD 'touch' has supported -d since FreeBSD 10 (2014). NetBSD has a -d option to its touch as well, but it expects a "human" format date & time, if the usage was: touch -md '2020-10-12 22:53:00Z' file it would work, but: touch -md 2020-10-12T22:53:00Z file touch: Could not parse `2020-10-12T22:53:00Z' With the 'T' in there, it is no longer a "human" format, but a standard one, and isn't recognised - I guess I could look at fixing parsedate(3) to handle that, but I'm not sure it is really warranted, parsedate() is a horrible ugly chunk of barely fuinctional code, and your updated patch will work fine. kre
On 10/17/20 1:46 AM, Robert Elz wrote:
touch -md 2020-10-12T22:53:00Z file touch: Could not parse `2020-10-12T22:53:00Z'
With the 'T' in there, it is no longer a "human" format, but a standard one, and isn't recognised - I guess I could look at fixing parsedate(3) to handle that, but I'm not sure it is really warranted, parsedate() is a horrible ugly chunk of barely fuinctional code, and your updated patch will work fine.
We went through *exactly* the same thought process in the Gnulib parse-datetime function, which is also a mess (perhaps even worse than NetBSD parsedate :-). However, in 2011 the POSIX requirement eventually got us to parse the form with the "T" in it, even though it's ugly and unreadable. I'm glad J.T. Conklin wrote the patch: https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=c2ecbc9a8262595b27f7... so that I didn't have to. By the way, the "human" format is specified in RFC 3339 section 5.6: https://tools.ietf.org/html/rfc3339#section-5.6 which allows a space. This was at my insistence. That "T" is bad for human readability, and readability is important, and if I had been part of the POSIX deliberations I would have insisted that it also allow " " instead of "T". GNU 'date' supports the "human" format as an extension to POSIX, and I assume NetBSD 'date' will continue to do so too.
Date: Sat, 17 Oct 2020 11:19:10 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <db471bb7-967f-7805-f850-cce37c93e098@cs.ucla.edu> | However, in 2011 the POSIX requirement Yes, saw that after I sent the previous message. So, I did write the code to handle it, and it is ugly (though could be uglier I suppose). Not yet committed to NetBSD, still pondering the real need (blind obedience to what POSIX mandates is not a goal of the project - there are several places where we simply refuse). | By the way, the "human" format is specified in RFC 3339 section 5.6: | | https://tools.ietf.org/html/rfc3339#section-5.6 Our "human format" is much more flexible than that, it allows stuff like "midnight tomorrow" or "three weeks ago" (or "3 weeks ago"), and further allows those kinds of things to be combined with more of the same ("next sunday 5 seconds ago") and/or with a more precise format (like the ISO variant, but also asctime output, rfc822 format, cvs format, ...) to set a base time rather than now (or some other input time_t), which is then modified. | and if I had been part of the POSIX deliberations I would have | insisted that it also allow " " instead of "T". It does. At least in touch: where: � YYYY are at least four decimal digits giving the year. � MM, DD, hh, mm, and SS are as with -t time. � T is the time designator, and can be replaced by a single <space>. � [.frac] and [,frac] are either empty, or a <period> ('.') or a <comma> (',') respectively, followed by one or more decimal digits, specifying a fractional second. � [tz] is either empty, signifying local time, or the letter 'Z', signifying UTC. If [tz] is empty, the resulting time shall be affected by the value of the TZ environment variable. Note the third bullet point. We extend that by allowing less than 4 Y digits, so (as meaningless as it is) '513-04-23 16:20:23Z' means some April date in the Dark Ages. We don't allow BC though... We also ignore the fraction (in all formats where it is permitted) - parsedate() is ancient, and returns a time_t which has no way to represent fractional seconds. One day I'll make a version which returns a bintime or a timespec or something. | GNU 'date' supports the "human" format as an extension to POSIX, | and I assume NetBSD 'date' will continue to do so too. Yes. That uses parsedate() as well. Several things do. kre
On 10/17/20 2:28 PM, Robert Elz wrote:
still pondering the real need (blind obedience to what POSIX mandates is not a goal of the project - there are several places where we simply refuse).
The real need in this case is supporting tzdb 2020c (:-). If POSIX requires a minor extension to NetBSD's current behavior, as is the case here, it should be an easy call anyway: just do it, might as well be compatible.
Our "human format" is much more flexible than that, it allows stuff like "midnight tomorrow" or "three weeks ago" (or "3 weeks ago"), and further
Yes, that's like Gnulib. The formats parse_datetime accepts are documented here: https://www.gnu.org/software/coreutils/manual/coreutils.html#Date-input-form...
| and if I had been part of the POSIX deliberations I would have | insisted that it also allow " " instead of "T".
It does. At least in touch:
Thanks, I missed that detail.
We extend that by allowing less than 4 Y digits, so (as meaningless as it is) '513-04-23 16:20:23Z' means some April date in the Dark Ages.
parse_datetime supports this syntax as well, though many filesystems don't work with those dates so 'touch' doesn't work properly. The Linux kernel tradition here seems to be to turn out-of-range timestamps into representable extrema, which is not a good thing if you ask me (should fail with EOVERFLOW).
We don't allow BC though...
parse_datetime doesn't go below year 0.
We also ignore the fraction (in all formats where it is permitted)
parse_datetime parses fractions down to nanosecond precision. It would be helpful if POSIX would standardize some of these extensions.
Date: Sat, 17 Oct 2020 18:32:39 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <9f551f2d-e4d6-97a8-f2f0-e3f83001542f@cs.ucla.edu> | The real need in this case is supporting tzdb 2020c (:-). We supported 2020c already without even noticing this issue, until I saw Deborah's message, and your reply, and I checked what our touch allows ... we don't use/generate/... rearguard, and we had already eliminated pacificnew in 2020b. | If POSIX requires a minor extension to NetBSD's current behavior, It is an ugly hackish extension - ugly because 'T' is also one of the Military type zone designators, and we support those as well (as part of supporting e-mail date formats). A date followed by 'T' might be either the first part of an ISO8601 format (or the restricted subset of that that posix expects support for) or a date in the UTC-0700 (or whichever 'T' is) timezone. All very ugly to deal with properly. If ISO had just used a '+' or '_' or something as the separator, it would have been both more language neutral ("time" (aka T) should be "Zeit" (aka Z) in German for example, right, and not an ascii character at all in Thai, Japanese, ...) and much simpler to deal with issues like these (which is why we supported the format with the space separator, but not with the 'T' and without writing any code to explicitly handle it, it was just a date in one of the common forms, a time in the most common form, and an optional 'Z' zone indicator - combine those together and the right thing just happens). | it should be an easy call anyway: just do it, might as well be compatible. So, not so easy, but it looks like the "just do it" will happen, especially since I wrote some code already. Mostly now just need to check to make sure nothing else broke while doing that. | parse_datetime supports this syntax as well, though many filesystems | don't work with those dates so 'touch' doesn't work properly. That's a different issue, and will go away once everyone is finally forced into abandoning all uses of 32 bit timestamps, which should finally happen by the end of this millennium. Posix doesn't require support for anything before 1970-01-01 (UTC) or after 2037-12-31 23:59:60 (if there happens to be a leap second then, 23:59:59 otherwise). Or maybe it is after 2038-01-01 00:00:00. | parse_datetime parses fractions down to nanosecond precision. If I add this (new function - though parseadate() would then just call it and ignore the fractional seconds in the result to generate a time_t) it would most likely be to generate a bintime, which allows down to 1/2^64 of a second (whatever that is in babble-speak - zetaseconds or something). | It would be helpful if POSIX would standardize some of these extensions. You can always submit a defect report, but you're not likely to get very far unless at least some of the commercial systems support it. What is really needed for POSIX is to have a better concept of optional features, "You don't have to support this, but if you do, this is the way it should be done" type stuff. That would allow a much simpler path to adding new facilities than presently exists. kre
On 10/18/20 2:23 AM, Robert Elz wrote:
If I add this (new function - though parseadate() would then just call it and ignore the fractional seconds in the result to generate a time_t) it would most likely be to generate a bintime, which allows down to 1/2^64 of a second (whatever that is in babble-speak - zetaseconds or something).
<https://man.netbsd.org/timeval.3> says struct bintime is meant to be used in the kernel only. I wonder how a nanosecond-resolution timestamp would be converted to bintime and back without rounding error, given that the usual tradition in timestamp conversion is to truncate toward minus infinity instead of rounding. The Gnulib time-parsing function sticks with struct timespec, since that's what POSIXish APIs give us.
Date: Sun, 18 Oct 2020 13:11:43 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <519bc685-8844-b341-31d3-91f5d8f2cb91@cs.ucla.edu> | <https://man.netbsd.org/timeval.3> says struct bintime | is meant to be used in the kernel only. Yes ... but nothing enforces that, and unenforced rules exist purely for the purposes of being broken, right? | I wonder how a nanosecond-resolution timestamp would | be converted to bintime and back without rounding error, The static inline functions used are in <sys/time.h> http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/sys/time.h?annotate=1.79&only_wi... but they don't. If you were to do that sequence you'd typically loose a ns each time around. So we don't do that, if we have timespecs we work in timespecs, converting them backwards and forwards would be foolish. For parsedate, we start with text, so we can generate anything. Depending upon the input, we may lose precision whatever we convert that into, if the text form has sufficient precision - converting something with 15 digits after the radix into a timespec you have to lose the low order). Generating the format with the greatest precision available means we're less likely to lose - to use it for touch we'll need to convert to a timespec (and for some filesystems perhaps eventually to a timeval - or for others even to a time_t) and we might lose some of the accuracy then, but parsedate() (or some new variant thereof) output can also be used for other things than feeding to filesystem sys calls. kre ps: here's a little userland (NetBSD buildable) program, with absolutely no error checking, which demonstrates the rounding issue. #include <sys/time.h> #include <stdlib.h> #include <stdio.h> int main(int argc, char *argv[]) { struct timespec ts, nts; struct bintime bt; ts.tv_sec = strtol(argv[1], 0, 0); ts.tv_nsec = strtol(argv[2], 0, 0); timespec2bintime(&ts, &bt); bintime2timespec(&bt, &nts); printf("%jd.%9.9ju -> %jx.%8.8jx -> %jd.%9.9ju", (intmax_t)ts.tv_sec, (intmax_t)ts.tv_nsec, (intmax_t)bt.sec, (intmax_t)bt.frac, (intmax_t)nts.tv_sec, (intmax_t)nts.tv_nsec); if (ts.tv_nsec != nts.tv_nsec) printf(" %jd ns change", (intmax_t)(nts.tv_nsec - ts.tv_nsec)); printf("\n"); return 0; }
On 10/18/20 2:10 PM, Robert Elz wrote:
Generating the format with the greatest precision available means we're less likely to lose
A better way to do it is to generate an internal format that tells you how much precision is in the timestamp. That way, the output of the date-parsing function gives you all the useful information of the input. Since text timestamps almost invariably use base 10 for subsecond information, the output of the timestamp parser should do likewise, and should say how many digits were in the input. At least, that's how I'd do it if I were extending a timestamp parser.
Date: Sun, 18 Oct 2020 14:37:57 -0700 From: Paul Eggert <eggert@cs.ucla.edu> Message-ID: <1437ae48-9d51-c956-9b77-11ae29cbe2a1@cs.ucla.edu> | A better way to do it is to generate an internal format that | tells you how much precision is in the timestamp. Yes, I'd considered that for other reasons in the past (primarily as a defence mechanism against people who insist in generating as much precision as they seem able to produce, in effect getting semi-random digits after a while). But creating yet another timestamp format isn't on my agenda right now. | Since text timestamps almost invariably use base 10 for | subsecond information, Yes, bintime was designed to deal with high res computer clocks, which are binary, not decimal (which is why the "in kernel only" remarks - by the time this info has escaped the kernel any high precision accuracy it once held has been lost in the random nature of the processing). | the output of the timestamp parser should do likewise, Yes, there's something in that, and it may be that I might use timespec instead of bintime for that reason. | and should say how many digits were in the input. but I doubt I would be inventing a new format to allow that, and if I were it would most probably be a bignum type format, so that it can represent arbitrary precisions and not be limited to the number of bits that were considered sufficient at the time the format was created. I think we have drifted a bit far off topic, and well out of the realm of civil timezones however. kre
participants (4)
-
Deborah Goldsmith -
Paul Eggert -
Paul Gilmartin -
Robert Elz