Re: Proposal for new ISO C 9x time API

Dear Timezone Gurus, I have now revised my proposal for a new <time.h> for C, which you can find on http://www.cl.cam.ac.uk/~mgk25/c-time/ When you have some time, please let me know what you think about it. Ideally we might manage to make this the basis for a proposal for ISO C9X. In case it is already too late for that, we might at least consider implementing it in the Olsen library and publish it independently (say via IETF as an RFC, or via the POSIX project). I have also added to the end of this page a pretty comprehensive list of links to online resources relevant to the topic, and a few more will come as soon as I have access to our scanner again. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
I have now revised my proposal for a new <time.h> for C, which you can find on
My initial impressions: I'm glad to see a new beginning in place of the "tmx" proposal, and acknowledgement of the problems it addressed poorly or not at all. I'm also glad to see this discussed in this forum. While this new proposal is much, much better than tmx, it still has some problems. They look fixable to me. First, it doesn't entirely solve the re-entrancy problem. If an error state and error message are to be carried around in the timezone_t object, then a "bad" timezone_t cannot be shared across threads which might have different locales. This part of the interface needs some rework. Given a bad timezone_t value, I don't see how strfxtime should indicate failure for those formats which use the time zone. An alternative interface, in place of tz_prep and tz_error, might be: timezone_t* tz_construct(const char *restrict tzstring, char *msg, int maxsize) which returns 0 for failure, and then if msg is non-null, stores a message into it up to max_size characters in length. This way there is never a bad timezone value to handle. (A null timezone is already specified to be treated like UTC.) Another problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it. This might best be another strfxtime directive. The names strfxtime, xtime_make, and xtime_breakup are inconsistent and not ideally chosen. May I suggest instead xtime_format: "To plan or arrange in a specified form" xtime_compose: "To make or create by putting together parts or elements." xtime_resolve: "To separate (something) into constituent parts." respectively? (Quotes are from the Random House unabridged dictionary.) I don't like to see the %H, %M, and %S formats restricted in their format to only '.' and ',' decimal separators. Separate directives (perhaps %.nH et al) that format only the fractional part would allow users to supply any decimal separator they chose, as in "%H:%M!%.3M" for "03:20!666". (I have seen satellite navigation systems with stranger choices of syntax.) Given that the format already specifies 64-bit operations on the more commonly-used component of the time, is there any reason to restrict the resolution of the fractional part to nanoseconds? Clock speeds greater than 1e9 Hz will be common before this interface comes into wide use. It may as well use (say) attoseconds, as in Bernstein's library. The library might also define constants corresponding to one nanosecond, microsecond, and millisecond in whatever unit is used for the fractional part, to minimize user errors. Typos: I believe the first paragraph describing representation of leap seconds refers to the member "sec" in one place where it should say "nsec". Also, the "note" text shows up in my browser in a microscopic font. I'm interested in what can be done to improve its suitability for incorporation into a future C++ standard. If the C and C++ bindings could be described simultaneously this would save a lot of trouble in the future. Nathan Myers ncm@cantrip.org

Nathan Myers wrote:
While this new proposal is much, much better than tmx, it still has some problems. They look fixable to me.
First, it doesn't entirely solve the re-entrancy problem. If an error state and error message are to be carried around in the timezone_t object, then a "bad" timezone_t cannot be shared across threads which might have different locales.
The tz_error function is explicitly allowed to generate its response based on the locale at the time tz_error is called. This eliminates the requirement to carry around the error message in the timezone_t object.
Another problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it. This might best be another strfxtime directive.
Timezone_t objects aren't necessarily constructed from Posix-style strings: they might use Olson-style long timezone names and an external database. If you mean simply returning the original second argument to tz_prep, I suppose that might serve some purpose.
Given that the format already specifies 64-bit operations on the more commonly-used component of the time, is there any reason to restrict the resolution of the fractional part to nanoseconds? Clock speeds greater than 1e9 Hz will be common before this interface comes into wide use.
UTC and TAI times are only correct to 100 ns or so. -- John Cowan http://www.ccil.org/~cowan cowan@ccil.org You tollerday donsk? N. You tolkatiff scowegian? Nn. You spigotty anglease? Nnn. You phonio saxo? Nnnn. Clear all so! 'Tis a Jute.... (Finnegans Wake 16.5)

Nathan Myers wrote on 1998-09-16 19:10 UTC:
While this new proposal is much, much better than tmx, it still has some problems. They look fixable to me.
First, it doesn't entirely solve the re-entrancy problem. If an error state and error message are to be carried around in the timezone_t object, then a "bad" timezone_t cannot be shared across threads which might have different locales. This part of the interface needs some rework. Given a bad timezone_t value, I don't see how strfxtime should indicate failure for those formats which use the time zone. An alternative interface, in place of tz_prep and tz_error, might be:
timezone_t* tz_construct(const char *restrict tzstring, char *msg, int maxsize)
which returns 0 for failure, and then if msg is non-null, stores a message into it up to max_size characters in length. This way there is never a bad timezone value to handle. (A null timezone is already specified to be treated like UTC.)
I expect tz_error usually to be called immediately after a tz_prep has signalled a problem. If programmers are sharing a bad timezone_t across thread, then honestly, that is their problem. Your concern sounds to me a bit far fetched and there are many obvious programming techniques to avoid the problem. There is no way in C for an API designer to enforce multi-threading safety, all we can do is to provide an API that enables multi-threading safe use of the functions, and that is IMHO good enough. I have indeed thought about a user provided finite buffer, as well as about tz_error doing a malloc. The main reason why I do not like both approaches is my recently gained experience with writing bindings from C libraries to other languages. Let's take Ada for example: All this mess with C returning variable length strings in a multi-threading safe way is a non-problem in Ada. Ada allows functions to return variable length arrays. The way most compilers (e.g., GNU Ada) do is this as follows: There is a secondary stack managed by the run-time library. Before an expression is evaluated which returns variable length arrays, the secondary stack pointer is saved. The space for the variable array to be returned is allocated on the secondary stack and can be used from there by other functions in the expression which use returned result. The secondary stack pointer is restored after the expression has been fully evaluated. If the returned variable length array has to be preserved for further use bejond the expression, it usually has to be copied (or relinked if the secondary stack uses the same storage pool with reference counters as the normal variable length string library). If I have an Ada function that returns the tz_error value, then I first have to call the C tz_error, then I have to find out how long the resulting string is (e.g., with strlen(), perhaps the length should also be returned), then I copy the string on the secondary stack of the Ada runtime-library and return from the function. In your proposal, I would have to introduce arbitrary limits for the length of the returnable string, which are difficult to justify to users of the API in other programming languages where such restrictions have no justification. Or I would have to iterate over the function that accepts a user-provided buffer, to find out how large this buffer has to be. In the end, considering interfacing to programming languages like Ada or Python which can comfortably return strings, I see my approach as preferably conceptually. In practice, error messages will hardly ever be longer than 80 characters, so a 256 limit for the maximum length should never hurt.
Another problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it. This might best be another strfxtime directive.
Which would mean that we have to force the implementor to store the original string. Is this really necessary? The user has provided the string himself, so why whould he depend on getting it back later. We certainly could easily add another strfxtime conversion specifier, but I wonder whether this is necessary at all.
I don't like to see the %H, %M, and %S formats restricted in their format to only '.' and ',' decimal separators. Separate directives (perhaps %.nH et al) that format only the fractional part would allow users to supply any decimal separator they chose, as in "%H:%M!%.3M" for "03:20!666". (I have seen satellite navigation systems with stranger choices of syntax.)
Thanks, that is a very good suggestion. I also have seem frequently h, m, or s in astronomical software as the separator instead of dot and comma, so the decimal separator should be user provided. Related problem: What is the semantics of decimal fractions of minutes and hours during a leap second. These are obviously illdefined and a neat solution is not possible (applications expecting leap seconds should never use decimal fractions of minutes, hours, and days). The best semantic I can thing of is to use max(nsec, 999_999_999) instead of nsec directly when calculating these decimal fractions. For decimal fractions of seconds, there is no problem, as long as we are with a leap second beyond 59 (which should be guaranteed unless someone introduces a UTC offset that is not an integral multiple of minutes).
Given that the format already specifies 64-bit operations on the more commonly-used component of the time, is there any reason to restrict the resolution of the fractional part to nanoseconds? Clock speeds greater than 1e9 Hz will be common before this interface comes into wide use. It may as well use (say) attoseconds, as in Bernstein's library.
I think attoseconds are horrible overkill. Considering precision: The best atomic clock on this planet (CS1 by PTB in Braunschweig) barely can do UTC with a real-time precision of one nanosecond. GPS provides to civilian users UTC with around 340 ns root mean square error, military users get down to perhaps tens on nanoseconds. Radio clocks like WWV or DCF77 provide UTC with around a millisecond precision, atmospheric path delays are often worse. NTP also works with millisecond precision under good conditions. So considering phase precision, we are many orders of magnitude better with nanoseconds than what is practically required. Considering resolution and uniqueness of local timestamps: It is hard to imagine that mass market silicon microprocessors will leave UHF and break through the 10 GHz barrier for internal clock speed during my lifetime. Reading out an internal monotonic clock counter, converting it to a portable UTC representation, and returning it via a system call interface will certainly take much longer than a few tens of instruction cycles (unless we see full hardware implementations of xtime_get(), for which I see no market justification), therefore processors that can do more than 10**9 calls to xtime_get() sound to me very much like science-fiction at the moment. Nanoseconds sound to me quite sufficient to guarantee unique timestamps with a comfortable safety margin. Considering frequency resolution: A nanosecond is also a nice representation for the phase and nanoseconds per second is a nice representation of the frequency of a kernel clock. If you add an adjustable real second every second to your phase base, then you can adjust the frequency with which your phase base progresses in nanoseconds per second, also known as parts per billion. This is significantly better already than the frequency change in your PC if you open the window and the temperature in the room drops a few Kelvin. Between these per-second adjustments, you do a linear extrapolation using a bus cycle counter and precalculated compensation factors. See the Linux kernel clock PLL for an implementation example, or Mill's papers that I quoted on my page.
The library might also define constants corresponding to one nanosecond, microsecond, and millisecond in whatever unit is used for the fractional part, to minimize user errors.
This is one solution. I would prefer another more general solution to minimize user error here: Allow underscores in numeric literals, like Ada does. I think 1_000_000_000 is much more readable than 1000000000 (it is really one billion?).
Typos: I believe the first paragraph describing representation of leap seconds refers to the member "sec" in one place where it should say "nsec".
I couldn't find this.
Also, the "note" text shows up in my browser in a microscopic font.
I only used the HTML <SMALL> tag and did not specify a specific font size. It is the reponsibility of your browser and its local configuration to select an adequate font size. If you use Netscape under X11, I can probably tell you how to fully configure all font sizes (look into Netscape.ad).
I'm interested in what can be done to improve its suitability for incorporation into a future C++ standard. If the C and C++ bindings could be described simultaneously this would save a lot of trouble in the future.
A C++ binding (and also an Ada95 binding) could use exceptions in order to signal the unavailablibity of a clock in xtime_get. This would leave the return value for the actual clock value. xtime would become a class under C++ and a private record under Ada, and the arithmetic functions and conversion functions to other existing time types would be appropriately overloaded. As I pointed out above, the tz_error message can be returned directly as an array under Ada (I don't think C++ has a comparable allocator-free mechanism). I don't see any immediate improvements that I could make to the C API to make it more suitable for bindings to more modern programming languages. I think though that this is a general design criterion, as the run-time libraries of most modern languages are sitting on top of some underlying C API. Therefore the underlying C API has to be as robust and flexible as possible. In order to not hinder proper implementations of higher language bindings. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Date: Fri, 02 Oct 1998 11:24:54 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> It is hard to imagine that mass market silicon microprocessors will leave UHF and break through the 10 GHz barrier for internal clock speed during my lifetime. Perhaps, but that doesn't mean that C apps won't run into sub-ns clocks soon. EE Times online reports that already people are doing 5 GHz in the labs with InP SRAM, with first commercialization rumored in the 2001 timeframe (ok, ok, the schedule may be marketing hype, but the underlying technology seems to be the real deal). NTT has a demo 350 GHz InP HEMT transistor that is an amusing feasibility study. I expect to see processors exceeding 1 THz in my lifetime (and I hope you live so long too :-). Reading out an internal monotonic clock counter, converting it to a portable UTC representation, and returning it via a system call interface One advantage of using a simple integer representation (instead of a struct) is performance: an implementer can avoid the conversion and system call overhead entirely, and compile the request for a timestamp into a direct access of the clock register. I expect tz_error usually to be called immediately after a tz_prep has signalled a problem. If programmers are sharing a bad timezone_t across thread, then honestly, that is their problem. Returning an integer value suitable as input to strerror would fix this problem in a thread-safe fashion, without having to use malloc. Related problem: What is the semantics of decimal fractions of minutes and hours during a leap second. These are obviously illdefined and a neat solution is not possible Why not? The day is just 1 second longer, so you use a denominator of 86401 instead of 86400 when computing the fraction. Of course you should use the larger denominator throughout the day, not just during the leap second.

Related problem: What is the semantics of decimal fractions of minutes and hours during a leap second. These are obviously illdefined and a neat solution is not possible
Why not? The day is just 1 second longer, so you use a denominator of 86401 instead of 86400 when computing the fraction.
(or 86399) Gulp! I did not catch this one, when I wrote yesterday that struct xtime can be adequate for inputting TIME_TAI values. Clearly, if you use a TIME_TAI value stored in struct xtime, you should not use fractional specifiers for strfxtime, because the result might be wrong if the corresponding period of time in the UTC scale include a leap second. Antoine

Antoine Leca wrote:
Clearly, if you use a TIME_TAI value stored in struct xtime, you should not use fractional specifiers for strfxtime, because the result might be wrong if the corresponding period of time in the UTC scale include a leap second.
Do the concepts of minutes, hours, days, etc. really make sense for TAI? -- John Cowan http://www.ccil.org/~cowan cowan@ccil.org You tollerday donsk? N. You tolkatiff scowegian? Nn. You spigotty anglease? Nnn. You phonio saxo? Nnnn. Clear all so! 'Tis a Jute.... (Finnegans Wake 16.5)

Paul Eggert wrote on 1998-10-02 11:43 UTC:
Related problem: What is the semantics of decimal fractions of minutes and hours during a leap second. These are obviously illdefined and a neat solution is not possible
Why not? The day is just 1 second longer, so you use a denominator of 86401 instead of 86400 when computing the fraction. Of course you should use the larger denominator throughout the day, not just during the leap second.
No, this doesn't work: All timestamps during the preceding day do not contain a marker that they are part of a 86401 seconds long day, therefore the calculation you propose is not possible because how would you know whether to divide by 86400 or by 86401. The best semantic for this I can think of (and that is what I suggest to use in strfxtime for calculating the fractional minutes and hours!) is to replace nsec by max(nsec, 999_999_999) in algorithms that convert to a presentation without leap second encoding. The Julian Date just stops during the leap second. I think there is nothing else you can do. A more general remark about your earlier proposal against using a struct for xtime: In any arithmetic time type you simply can't insert leaps seconds properly. In order to insert leap seconds, you have to break down time into components and let the smaller component overflow. This is done in struct tm and on the UTC clock with the famous sec=60 overflow and it is also done in struct xtime with the nsec=1_000_000_000 overflow. The whole beauty of struct xtime compared to any proposal of using just an arithmetic time (essentially time_t again) is exactly that it is a time that is broken down at the second level such that leap seconds can be inserted by overflow of one component. In UTC, the minute flows over, in struct xtime, the second flows over. Having a time type that allows to represent leap seconds by internal overflow and having separate UTC, monotonic, and perhaps also TAI clocks is the very crux of getting rid of the necessity of leap second tables. You might call it state-less time, because there is no state such as a leap second table or a flag that a leap second is nearby involved in any of this. [BTW: An interesting, though more academic side note: since leap seconds can only appear at the end of minutes, it is a not too far fetched idea to break down time into a 32-bit minute counter since the epoch and a nanominute (nmin) counter which can overflow to 1 1/60 minutes in the case of a leap second. 32-bit minute counters cover a range of 9.8 kiloyears, which is quite reasonable as it covers most of the recorded human history. Equivalently, since leap seconds can only appear at the end of a UTC day, a MJD day and nanoday counter would be even better. A nanoday is 86.6 µs, which might already be a bit long for unique timestamps per server.] Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Date: Fri, 02 Oct 1998 18:08:13 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk>
What is the semantics of decimal fractions of minutes and hours during a leap second. These are obviously illdefined and a neat solution is not possible
Why not? The day is just 1 second longer, so you use a denominator of 86401 instead of 86400 when computing the fraction. Of course you should use the larger denominator throughout the day, not just during the leap second.
No, this doesn't work: All timestamps during the preceding day do not contain a marker that they are part of a 86401 seconds long day, They don't have to contain a marker. You can figure it out by looking at what day the timestamp is in, and then consulting the leap second table (and time zone table, of course). Apparently one if your goals is that strftime should not have to consult a leap second table, even on systems that support leap seconds. I don't see why this is a desirable goal. In any arithmetic time type you simply can't insert leaps seconds properly. Obviously you need a boolean as well as the arithmetic time value, to denote the fact that the time stamp is within a leap second. The question is: how do you encode the boolean? The struct xtime proposal encodes the boolean as part of an out-of-range sub-second value. But I think it's more straightforward to encode it as a boolean. Having a time type that allows to represent leap seconds by internal overflow and having separate UTC, monotonic, and perhaps also TAI clocks is the very crux of getting rid of the necessity of leap second tables. I don't understand this point. First, you cannot get rid of leap second tables if you want to convert properly between UTC and the other formats, on systems that support leap seconds. Second, if you represent a UTC timestamp with an integer plus a boolean leap-second flag, you have exactly the same information that you would have with struct xtime (assuming XTIMES_PER_SEC is 1000000000). So I don't see why one system is any better than the other in terms of avoiding leap second tables. I agree that the C standard should allow sloppy implementations that don't know about leap seconds; those implementations won't need leap second tables at all, of course.

Paul Eggert wrote on 1998-10-02 19:10 UTC:
Why not? The day is just 1 second longer, so you use a denominator of 86401 instead of 86400 when computing the fraction. Of course you should use the larger denominator throughout the day, not just during the leap second.
No, this doesn't work: All timestamps during the preceding day do not contain a marker that they are part of a 86401 seconds long day,
They don't have to contain a marker. You can figure it out by looking at what day the timestamp is in, and then consulting the leap second table (and time zone table, of course).
Apparently one if your goals is that strftime should not have to consult a leap second table, even on systems that support leap seconds. I don't see why this is a desirable goal.
One of the fundamental concepts behind my API is exactly to make sure that there are only two functions, which would access to a leap second table, and both are optional, that is are always allowed to return that they have no information available. These two functions are xtime_conv() and tz_jump(). They allow you to access and iterate trough a leap second table. All other functions are *never* expected to access a leap second table (and this includes xtime_add and xtime_diff, to be added soon). If you think that any of the other functions need access to a leap second table, then you probably haven't yet understood the fundamental design principle behind my API. Why? In my opinion, leap second tables are a very dubious and extremely dangerous concept. In real life, they are practically guaranteed to be out-of-date on most systems and if we build leap-second tables into the architecture of the API as a fundamental and not as an auxiliary element, then I see already a long list of comp.risks postings coming in the next decades which report about bad things that had happened in distributed systems due to inconsistent and out-of-date leap-second tables, and what fools the designer of the C time API have been to introduce such an irresponsible concept. Therefore I want to have the leap second table access strictly restricted to the two auxiliary functions xtime_conv and tz_jump.
In any arithmetic time type you simply can't insert leaps seconds properly.
Obviously you need a boolean as well as the arithmetic time value, to denote the fact that the time stamp is within a leap second. The question is: how do you encode the boolean? The struct xtime proposal encodes the boolean as part of an out-of-range sub-second value. But I think it's more straightforward to encode it as a boolean.
I *have* played with this idea and written example pseudo-code for both nsec overflow and boolean leap second indicators. I found the nsec overflow to be slightly simpler to implement and slighly more intuitive. In addition, it does not waste any bits if only non-UTC clocks are used. I agree that it is possible to define an alternative struxt xtime that consists of just an arithmetic time scale without an encoding for inserted leap seconds, plus a separate boolean flag (at least 8 bits that mess up word alignment!) that indicates a leap second and uses the encoding values of the preceeding or following non-leap second to represent the leap second. Another idea I have played around with is to use the LSB in the arithmetic type. I was not too happy with any of the results of these gedankenexperiments. The nsec overflow idea is not just a quick first idea: I have considered many alternatives and in order to convince me that my considerations where wrong you will certainly have to present me with a good discussion of the detailed implementation advantages of your encoding versus mine, because I do see many disadvantages of your system compared to mine.
Having a time type that allows to represent leap seconds by internal overflow and having separate UTC, monotonic, and perhaps also TAI clocks is the very crux of getting rid of the necessity of leap second tables.
I don't understand this point. First, you cannot get rid of leap second tables if you want to convert properly between UTC and the other formats, on systems that support leap seconds.
As I said before, I want to restrict the access to the dangerous leap second tables to two functions. If programming guide lines in a project forbid any semantic dependency on leap seconds, then to verify this guideline, in my API you only have to study calls to xtime_conv() and tz_jump() in order to understand potential effects of out-of-date leap-seconds tables. In the other proposals for handling leap seconds that I have seen, almost everything depends on correct leap second tables (especially in the "let's only use TAI" proposals), and it is almost impossible to avoid semantic dependence on them. This can have serious saftey and security implications IMHO.
Second, if you represent a UTC timestamp with an integer plus a boolean leap-second flag, you have exactly the same information that you would have with struct xtime (assuming XTIMES_PER_SEC is 1000000000). So I don't see why one system is any better than the other in terms of avoiding leap second tables.
It is certainly identical in terms of separating the dependency of leap-second tables. Main advantages of my representation are: I utilize the memory layout of the struct better, I enable compatibility with POSIX struct timespec, and I think the example code I have played around with looks neater.
I agree that the C standard should allow sloppy implementations that don't know about leap seconds; those implementations won't need leap second tables at all, of course.
Systems with and without leap second tables should only differ in their semantic in a small and well-defined way, and I think my proposal provides exactly this. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
In my opinion, leap second tables are a very dubious and extremely dangerous concept.
Leap seconds are an inherent part of UTC. If you have an accurate UTC display, and an accurate difftime(), then you have a leap-second table. ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

Date: Sun, 04 Oct 1998 11:16:52 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> One of the fundamental concepts behind my API is exactly to make sure that there are only two functions, which would access to a leap second table, and both are optional, that is are always allowed to return that they have no information available. These two functions are xtime_conv() and tz_jump(). Sorry, I don't follow you here. In your spec, if xtime_get with TIME_UTC returns a time with nsec>=1000000000, then the application has obtained a leap second; so xtime_get's implementation must have accessed a leap second table of _some_ sort. Normally, one would think that an implementation has only two choices: (1) Support leap seconds (e.g. the Olson code in "right" mode, or Bernstein's library); or (2) Omit leap seconds (e.g. POSIX.1). It sounds like you're trying to allow for another possibility: (3) Support just some leap seconds, without having a complete leap second table. For example, it sounds like you want to cater to implementations that know only the leap second nearest to the present, or something like that. The leap second table is _always_ incomplete, of course, since we don't know all future leap seconds; so this problem of incomplete knowledge of leap seconds is inherent to any high-quality interface. So I suggest that we give the programmer a way to access to the entire leap second table known to the implementation. E.g. if the implementation knows only the next leap second, then it would return ``I don't know'' for questions about later (or previous) leap seconds. This seems to me to be the most natural way to model implementations of type (1), (2) and (3). By the way, I'm not familiar with type (3) C implementations in practice -- which ones are you thinking of? It would be helpful to have a bit more familiarity with the real-world issues here. (Also, this issue needs to be discussed better in the rationale!) leap second tables are a very dubious and extremely dangerous concept. A implementation must have a leap-second table of _some_ sort, if only a partial one, if it wants to support leap seconds; otherwise, xtime_get with TIME_UTC can't return leap seconds. I found the nsec overflow to be slightly simpler to implement and slighly more intuitive. In addition, it does not waste any bits if only non-UTC clocks are used. No, struct xtime normally wastes more than two bits, since it puts a nanosecond value (< 1000000000) into a member that can store numbers more than twice as large (< 2147483648), and also wastes a sign bit. Even with TIME_UTC, struct xtime wastes more than one bit. Here's an encoding that uses about .00001 bit per timestamp, assuming an integer counter is used for timestamps: If T is a TIME_UTC timestamp, it identifies the point of time that is T%D xtime intervals after T/D UTC days after the epoch, where D is 86401*XTIMES_PER_SEC. For space-saving this beats all other encodings proposed so far. In the other proposals for handling leap seconds that I have seen, almost everything depends on correct leap second tables (especially in the "let's only use TAI" proposals) I also would not like to require correct leap second tables. But I hope that we can do better than the current proposal. I think the example code I have played around with looks neater. It would be helpful if you could publish the sample code, when it's ready. I have considered many alternatives I know, but we haven't exhausted the alternatives yet! A lot more thinking needs to be done with this proposal.

Paul Eggert wrote on 1998-10-05 22:23 UTC:
It sounds like you're trying to allow for another possibility:
(3) Support just some leap seconds, without having a complete leap second table.
For example, it sounds like you want to cater to implementations that know only the leap second nearest to the present, or something like that.
Yes. Option (3) is in real life the by far most frequent and most realistic possibility. There are hundreds of thousands of NTP controlled workstations out there that do exactly this. Most time services provide you an announcement of the next leap second. For instance DCF77 (the German 77.5 kHz time service for which you can buy 20 dollar serial port receivers and 50 dollar radio controlled digital wrist watches that work all over Central Europe) sets a leap second warning bit during the 60 minutes preceding the leap second. GPS announces a few weeks ahead of time the time when the next leap second occurs, and NTP does something similar. If you want to call a system state that knows only about the next leap seconds in order to roll correctly through it a system that "has a leap second table", well, ok. All that these systems know is that there will be one leap second and no other leap second until then. The existing time distribution infrastructure announces the next leap second in order to prevent a loss of synchronization right after the leap second, but it was not designed to distribute information about the history of TAI versus UTC relationships. That is what I repeated several times that UTC is extremely widely available but TAI is not, and that this is the reason why I consider it difficult to propose a robust implementation that is entirely based on TAI. It is important to understand that (except GPS) most time services announce only that there will be a leap second, but not what the current and coming UTC-TAI offset is!
The leap second table is _always_ incomplete, of course, since we don't know all future leap seconds; so this problem of incomplete knowledge of leap seconds is inherent to any high-quality interface. So I suggest that we give the programmer a way to access to the entire leap second table known to the implementation.
My API does this in form of tz_jump.
E.g. if the implementation knows only the next leap second, then it would return ``I don't know'' for questions about later (or previous) leap seconds.
Done.
This seems to me to be the most natural way to model implementations of type (1), (2) and (3).
Done.
By the way, I'm not familiar with type (3) C implementations in practice -- which ones are you thinking of?
Some example code from the standard Red Hat Linux kernel running on my machine here right now (and running on around 5 million other PCs, active on those hundreds of thousands of PCs that are NTP synchronized): /usr/src/linux/kernel/sched.c: ------------------------------------------------------------------------- /* * this routine handles the overflow of the microsecond field * * The tricky bits of code to handle the accurate clock support * were provided by Dave Mills (Mills@UDEL.EDU) of NTP fame. * They were originally developed for SUN and DEC kernels. * All the kudos should go to Dave for this stuff. * */ static void second_overflow(void) { long ltemp; /* Bump the maxerror field */ time_maxerror += time_tolerance >> SHIFT_USEC; if ( time_maxerror > MAXPHASE ) time_maxerror = MAXPHASE; /* * Leap second processing. If in leap-insert state at * the end of the day, the system clock is set back one * second; if in leap-delete state, the system clock is * set ahead one second. The microtime() routine or * external clock driver will insure that reported time * is always monotonic. The ugly divides should be * replaced. */ switch (time_state) { case TIME_OK: if (time_status & STA_INS) time_state = TIME_INS; else if (time_status & STA_DEL) time_state = TIME_DEL; break; case TIME_INS: if (xtime.tv_sec % 86400 == 0) { xtime.tv_sec--; time_state = TIME_OOP; printk("Clock: inserting leap second 23:59:60 UTC\n"); } break; case TIME_DEL: if ((xtime.tv_sec + 1) % 86400 == 0) { xtime.tv_sec++; time_state = TIME_WAIT; printk("Clock: deleting leap second 23:59:59 UTC\n"); } break; case TIME_OOP: time_state = TIME_WAIT; break; case TIME_WAIT: if (!(time_status & (STA_INS | STA_DEL))) time_state = TIME_OK; } /* * Compute the phase adjustment for the next second. In * PLL mode, the offset is reduced by a fixed factor * times the time constant. In FLL mode the offset is * used directly. In either mode, the maximum phase * adjustment for each second is clamped so as to spread * the adjustment over not more than the number of * seconds between updates. */ if (time_offset < 0) { ltemp = -time_offset; if (!(time_status & STA_FLL)) ltemp >>= SHIFT_KG + time_constant; if (ltemp > (MAXPHASE / MINSEC) << SHIFT_UPDATE) ltemp = (MAXPHASE / MINSEC) << SHIFT_UPDATE; time_offset += ltemp; time_adj = -ltemp << (SHIFT_SCALE - SHIFT_HZ - SHIFT_UPDATE); } else { ltemp = time_offset; if (!(time_status & STA_FLL)) ltemp >>= SHIFT_KG + time_constant; if (ltemp > (MAXPHASE / MINSEC) << SHIFT_UPDATE) ltemp = (MAXPHASE / MINSEC) << SHIFT_UPDATE; time_offset -= ltemp; time_adj = ltemp << (SHIFT_SCALE - SHIFT_HZ - SHIFT_UPDATE); } /* * Compute the frequency estimate and additional phase * adjustment due to frequency error for the next * second. When the PPS signal is engaged, gnaw on the * watchdog counter and update the frequency computed by * the pll and the PPS signal. */ pps_valid++; if (pps_valid == PPS_VALID) { pps_jitter = MAXTIME; pps_stabil = MAXFREQ; time_status &= ~(STA_PPSSIGNAL | STA_PPSJITTER | STA_PPSWANDER | STA_PPSERROR); } ltemp = time_freq + pps_freq; if (ltemp < 0) time_adj -= -ltemp >> (SHIFT_USEC + SHIFT_HZ - SHIFT_SCALE); else time_adj += ltemp >> (SHIFT_USEC + SHIFT_HZ - SHIFT_SCALE); /* in the NTP reference this is called "hardclock()" */ static void update_wall_time_one_tick(void) { /* * Advance the phase, once it gets to one microsecond, then * advance the tick more. */ time_phase += time_adj; if (time_phase <= -FINEUSEC) { long ltemp = -time_phase >> SHIFT_SCALE; time_phase += ltemp << SHIFT_SCALE; xtime.tv_usec += tick + time_adjust_step - ltemp; } else if (time_phase >= FINEUSEC) { long ltemp = time_phase >> SHIFT_SCALE; time_phase -= ltemp << SHIFT_SCALE; xtime.tv_usec += tick + time_adjust_step + ltemp; } else xtime.tv_usec += tick + time_adjust_step; if (time_adjust) { /* We are doing an adjtime thing. * * Modify the value of the tick for next time. * Note that a positive delta means we want the clock * to run fast. This means that the tick should be bigger * * Limit the amount of the step for *next* tick to be * in the range -tickadj .. +tickadj */ if (time_adjust > tickadj) time_adjust_step = tickadj; else if (time_adjust < -tickadj) time_adjust_step = -tickadj; else time_adjust_step = time_adjust; /* Reduce by this step the amount of time left */ time_adjust -= time_adjust_step; } else time_adjust_step = 0; } -------------------------------------------------------------------------
It would be helpful to have a bit more familiarity with the real-world issues here.
(Also, this issue needs to be discussed better in the rationale!)
I have provided references to papers by Dave Mills at the end of the document that discuss the kernel treatment of leap seconds in some detail. Linux is one system where you can see the mechanics yourself in the source code. I think DEC does something similar, but I don't have access to source or documentation. All the ideas that I presented are not just some academic gobbledeegoop, but are based on half a decade of operational implementation experience. I was a student of Dr. Frank Kardel at the University of Erlangen, who was one of the xntpd maintainers in the early 1990s and who demonstrated successful NTP-based microsecond kernel PLL clock synchronization of SunOS servers. I "grew up" in a research lab where we routinely observed the cycles of the air conditioner in the server room my measuring the temperature induced phase drifts of the clock generators in Sun servers via suitable software and time protocols. Accusation made my others about my possible ignorance with regard to software timing are a bit insulting. It is just difficult to communicate my background in this field in a formal proposal. You are probably right if you mean that the ISO C committee members are unlikely to be familiar with mechanisms, which have been very common knowledge for NTP hackers for half a decade. It would be fortunate if I could personally attend a meeting and present these issues myself, but I don't have the funding for such a trip (unless they do one of their next meetings in London). If I wrote down all background information that I took into consideration, it would probably become a book (may be, it will one day ...;-).
leap second tables are a very dubious and extremely dangerous concept.
A implementation must have a leap-second table of _some_ sort, if only a partial one, if it wants to support leap seconds; otherwise, xtime_get with TIME_UTC can't return leap seconds.
Ok, let me formulate this clearer: My concern with proposals such as libtai is the following: Systems receive UTC from the outside (a fact of life). Then Dan Bernstein's systems transform the received UTC into TAI and use TAI for all further processing (including in communication protocols). In order to do the UTC->TAI conversion correctly, you have to know not only when the next leap second is, but also the CORRECT NUMBER of leap seconds since 1972, since the current TAI offset is NOT announced in almost all time services (and I have been complaining about this for half a decade now to the operators of time services, so please don't blame me that it is this way!). Each time one of Dan Bernstein's systems misses a leap second announcement, the TAI value that this system uses from then on will be off by one more second. The TAIs used in a distributed system will start to move apart, until someone uploads fresh current TAI-UTC offsets into all machines (and I certainly don't have to tell people in this forum how "eager" normal users are with keeping their configuration tables up to date).
Here's an encoding that uses about .00001 bit per timestamp, assuming an integer counter is used for timestamps:
If T is a TIME_UTC timestamp, it identifies the point of time that is T%D xtime intervals after T/D UTC days after the epoch, where D is 86401*XTIMES_PER_SEC.
Ok, so you reserve values for a leap second at the end of every UTC day (in effect making every day 86401 SI seconds long on the scale). If you go through the comp.protocols.time.ntp archives, then you will find that I proposed exactly this encoding (plus a few variants) years ago. I am not that enthusiastic any more about it, because if complicates many algorithms significantly, it was more intended as a compatibility hack with the old C89 spec. It would also completely mess up the elegance of my API in that suddenly strfxtime and xtime_make/breakup would have to know from what type of clock this timestamp came: Note that in my encoding, the two times 1972-01-01 00:00:00 UTC and 1972-01-01 00:00:00 TAI (which are 10 s apart) are represented by the same xtime value. An xtime value is just seen as an encoding for a year/month/day/hour/minute/second representation, and any routine that wants to convert these xtime values into broken-down values does not have to knwo whether this is a UTC or TAI (or even a local time) value. With your alternative, this would not work any more, unless you use 86401 seconds per day for both UTC and TAI (which your clause "If T is a TIME_UTC timestamp" does not suggest).
For space-saving this beats all other encodings proposed so far.
Two-bit space-saving at the cost of the complexity of algorithms (= likelihood of user missunderstandings) does not sound attractive to me. I think, the POSIX timespec got it almost right, and I see no reason why we shouldn't simply build on that encoding. The only doubts that I have about xtime is that I think it is with 96 bits a bit too long. I would feel slightly better with 64 bits. I am of course also aware of the DCE and CORBA encodings (see the reference section of my proposal), which use a 64-bit 0.1 µs counter since 1601-01-01. It is a nice idea and 100 ns is probably still more than sufficient for practically all applications, but like the old POSIX time_t, it is not broken-down and therefore does not provide for a nice leap second encoding (unless we use the above 1 day + 1 sec hack). So 96-bit still seems to be the better compromise to me, and in the ned I think that having a cosmological range won't hurt. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
Each time one of Dan Bernstein's systems misses a leap second announcement, the TAI value that this system uses from then on will be off by one more second.
Correct. This means that local times will still be displayed correctly, but time differences across N missed leap seconds will be off by N. In other words: ``Without a leap-second table, libtai performs just as badly as Markus's proposal.'' ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

"D. J. Bernstein" wrote on 1998-10-06 17:16 UTC:
Markus Kuhn writes:
Each time one of Dan Bernstein's systems misses a leap second announcement, the TAI value that this system uses from then on will be off by one more second.
Correct. This means that local times will still be displayed correctly, but time differences across N missed leap seconds will be off by N.
What concerns me is: If you use these erroneous TAI timestamps in communication protocols and in a distributed system inconsistent TAI-UTC tables are used on the various nodes to convert back from TAI via UTC to local time, then you will get wrong local times displayed across the system. If you compare TAI values in system logs across systems, you will get inconsistencies (e.g., negative network delays, etc.) => better use UTC.
In other words: ``Without a leap-second table, libtai performs just as badly as Markus's proposal.''
First, I have just posted the 3-line code of how you can use xtime_conv() and xtime_diff in my API to determine the correct number of SI seconds between timestamps, so my proposal performs absolutely correctly despite your continued attempts of misleading rhetoric. Secondly, in my API the user has a much better chance to find out about the lack of an up-to-date leap second table, therefore the opposite statement that in the worst case my proposal performs as badly as yours is technically probably a bit more justified. I hope you agree that I have shown how your libtai API can be implemented in a portable way on top of my xtime proposal without loss of functionality.
Ever heard of TCP/IP? How about UNIX?
Well, I vaguely remember having heard about these things once or twice, but feel free to give me an introduction, since I am quite new to all these confusing acronyms ...
How about system logs? Timestamps in logs aren't just for clock displays; they're also used for profiling and accounting. Time differences are crucial.
Interesting point: I have been told by a phone exchange designer whom I asked about their timing hardware, that in the telephone industry, it is common practice not to charge for the traffic during leap seconds. Most digital phone systems run phase locked with UTC today. I think they have survived their financial losses of around 0.0000021 percent caused by leap seconds quite well. Check your contracts to make sure you are not calculating unrealistic communications costs if you your TAI for such applications. Similar things apply to power supply companies, although I am not sure about the details. If I remember correctly, Tanenbaum writes in his Distributed Systems book that US power companies provide you every UTC day 86400*60 power cycles, and that they reduce the frequency to 59 Hz for one minute in the case of an inserted leap second. So for phone charges and AC power cycles, my xtime_diff on TIME_UTC provides quite the right semantics. Nice, isn't it? The real world is ticking in UTC (except of course navigation systems, and even there GLONASS is ticking in UTC, while GPS is locked to TAI).
Markus Kuhn writes:
and if you think that the Pentium's internal bus cycle counter
Do you have any idea what you're talking about?
I very much like to think so.
The bus frequency is much lower than the CPU frequency.
Sure. I think I used technically very precise language and referred clearly and correctly to the "internal bus", in other words to the on-chip bus that runs with the internal CPU clock frequency. You will need a microprobing needle or (especially for the new Pentium) an EBT in order to make measurements on it. Been there, done that. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
First, I have just posted the 3-line code of how you can use xtime_conv() and xtime_diff in my API to determine the correct number of SI seconds between timestamps,
To make that work correctly, you need an up-to-date leap-second table.
Secondly, in my API the user has a much better chance to find out about the lack of an up-to-date leap second table,
Programs that care about the table can see it with either API. Programs that simply want the most accurate available clock information can get it with my API. You force them to jump through an unnecessary series of hoops to obtain extra information that they simply throw away. ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

On Tue, 06 Oct 1998 13:55:06 +0100, Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> wrote:
It would also completely mess up the elegance of my API in that suddenly strfxtime and xtime_make/breakup would have to know from what type of clock this timestamp came:
Note that in my encoding, the two times
1972-01-01 00:00:00 UTC
and
1972-01-01 00:00:00 TAI
(which are 10 s apart) are represented by the same xtime value. An xtime value is just seen as an encoding for a year/month/day/hour/minute/second representation, and any routine that wants to convert these xtime values into broken-down values does not have to knwo whether this is a UTC or TAI (or even a local time) value. With your alternative, this would not work any more, unless you use 86401 seconds per day for both UTC and TAI (which your clause "If T is a TIME_UTC timestamp" does not suggest).
How can strfxtime() determine what clock the xtime value being passed refers to? For the time 1972-01-01 00:00:00 UTC, we have the TIME_UTC value {86400*365*2,0} and the TIME_TAI value {86400*365*2+10,0} (per the definition of TIME_UTC and TIME_TAI in the documetation of the xtime_get() function). These are different, so I fail to see how strfxtime() can avoid having knowledge of which clock the xtime value refers to. When I read the proposal I presumed that only TIME_UTC values should be passed to strfxtime() (whether the result of the appropriate xtime_get() or xtime_conv() call). I now feel that the failure to explictly state that strfxtime()'s xtime argument should be a TIME_UTC value is a bug in the documentation of the strfxtime() function. --Ken Pizzini

Ken Pizzini wrote on 1998-10-07 07:09 UTC:
When I read the proposal I presumed that only TIME_UTC values should be passed to strfxtime() (whether the result of the appropriate xtime_get() or xtime_conv() call).
That is correct if you want strfxtime() to convert to some local time, because the timezone_t objects convert from UTC to a local time and therefore have to be fed with UTC and not with TAI if you want correct local times. The same applies to xtime_breakup(). There is however one exception: If you have a TAI timestamp and what to print it as a TAI timestamp, then you just pass to strfxtime the TAI timestamp and specify NULL as the time zone. NULL means that the output shall be a "UTC time", but since the relationship between xtime and broken-down time is identical for both TAI and UTC, you will get the correct TAI output as well with NULL. Therefore, strfxtime can also be passed TAI timestamps with a NULL timezone_t in order to print correct TAI timestamps and strftime() does not even have to know about that this is a TAI timestamp. Note that the semantics of (timezone_t *) NULL is the only timezone whose semantics is precisely specified in the proposal by providing example xtime_make() C code (not yet completed). The semantics of the other time zones is only specified indirectly in that they should be handled in analogy to how UTC broken-down time is handled, plus some (still to be added) remarks about the purpose of tm_isdst. This will allow for all sorts of bizarre timezone definitions, including (of course hypothetical) timezones based on UT1 (if an internal UT1 table is available), or even timezones on other planets, just to document the flexibility of the API. Example: if (xtime_get(&t, TIME_UTC) & TIME_UTC) strfxtime(..., "%H:%M:%S UTC", &t, NULL); if (xtime_get(&t, TIME_TAI) & TIME_TAI) strfxtime(..., "%H:%M:%S TAI", &t, NULL); I think this is a much nicer embedding of both UTC and TAI into the scheme of things than having to pass on to xtime_breakup(), xtime_make(), and strfxtime() a separate parameter that tells whether we are using a UTC or a TAI timestamp.
I now feel that the failure to explictly state that strfxtime()'s xtime argument should be a TIME_UTC value is a bug in the documentation of the strfxtime() function.
I thought it is very clearly stated in the definition of what a timezone_t object is doing that it converts between UTC xtime timestamps and some broken-down local time, and the xtime_breakup(), xtime_make(), and strfxtime() use the timezone_t objects exactly this way. I will add an informal note to strfxtime() to remind the reader about this definition. Just explicitely stating "that strfxtime()'s xtime argument should be a TIME_UTC value" would not be accurate, because as you see above, with a (timezone_t *) NULL parameter, passing TIME_TAI values also makes a lot sense. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> wrote:
If you have a TAI timestamp and what to print it as a TAI timestamp, then you just pass to strfxtime the TAI timestamp and specify NULL as the time zone. NULL means that the output shall be a "UTC time", but since the relationship between xtime and broken-down time is identical for both TAI and UTC, you will get the correct TAI output as well with NULL. Therefore, strfxtime can also be passed TAI timestamps with a NULL timezone_t in order to print correct TAI timestamps and strftime() does not even have to know about that this is a TAI timestamp.
I'm sorry, I still fail to follow this. 1972-01-01 00:00:00 UTC has a TIME_UTC xtime of {86400*365*2,0} and TIME_TAI has an xtime of {8600*365*2+10,0}, and TIME_UTC counts non-leapseconds while TIME_TAI counts all seconds, including leapseconds. So the xtime values for the same instant in time on these two clocks will always differ by (TAI-UTC) seconds, and values on these two clocks will differ for the same broken-down time representation (except perhaps for the brief period of history when TAI==UT1 held, if you want to speculate on the pre-1972 relationship between TAI and "UTC"). --Ken Pizzini

Ken Pizzini wrote on 1998-10-07 09:12 UTC:
Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> wrote:
If you have a TAI timestamp and what to print it as a TAI timestamp, then you just pass to strfxtime the TAI timestamp and specify NULL as the time zone. NULL means that the output shall be a "UTC time", but since the relationship between xtime and broken-down time is identical for both TAI and UTC, you will get the correct TAI output as well with NULL. Therefore, strfxtime can also be passed TAI timestamps with a NULL timezone_t in order to print correct TAI timestamps and strftime() does not even have to know about that this is a TAI timestamp.
I'm sorry, I still fail to follow this. 1972-01-01 00:00:00 UTC has a TIME_UTC xtime of {86400*365*2,0} and TIME_TAI has an xtime of {8600*365*2+10,0}, and TIME_UTC counts non-leapseconds while TIME_TAI counts all seconds, including leapseconds. So the xtime values for the same instant in time on these two clocks will always differ by (TAI-UTC) seconds, and values on these two clocks will differ for the same broken-down time representation.
Exactly. As they are supposed to be! Look: If we place a UTC and a TAI clock next to each other, we see the following at this point in time: 1972-01-01 00:00:00 UTC = 1972-01-01 00:00:10 TAI If you feed {86400*365*2,0} (the UTC value) into strfxtime you get 1972-01-01 00:00:00 as output (the correct UTC clock display) and if you feed {8600*365*2+10,0} (the corresponding TAI value at the same time) into it, you get 1972-01-01 00:00:10 as output (the correct TAI clock display at the same time). So everything works out exactly right. (With (timezone_t *) NULL in each case.) Wonderful, isn't it? May be you just didn't understand that a TAI clock does indeed show a different time than a UTC clock? Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Date: Wed, 07 Oct 1998 10:02:04 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> If you have a TAI timestamp and what to print it as a TAI timestamp, then you just pass to strfxtime the TAI timestamp and specify NULL as the time zone. First, this doesn't always work under the current struct xtime proposal. E.g. strfxtime with a NULL time zone expands "%Z" to something like "UTC", which isn't correct for TAI timestamps. Second, even assuming the users avoids buggy formats like "%Z", the resulting information is useless in practice. Who wants broken-down TAI times? At bottom, TAI is an integer, not a Gregorian calendar date-time indication. It is much more useful to convert the TAI timestamps to broken-down civil times. Third, I agree that for TIME_LOCAL, strfxtime etc. return useful information in most cases, but the buggy formats like %Z are still buggy. This should get fixed. Fourth, for TIME_MONOTONIC, TIME_PROCESS, or TIME_THREAD, strfxtime etc. return garbage, no matter how you slice it. So more work is needed here; this area is too confusing and error-prone. I think this is a much nicer embedding of both UTC and TAI into the scheme of things than having to pass on to xtime_breakup(), xtime_make(), and strfxtime() a separate parameter that tells whether we are using a UTC or a TAI timestamp. You don't need to pass a clock type to those functions to get things to work correctly; all you need to do is pass a clock type to tz_prep. tz_prep can store the clock type information somewhere in the timezone_t value.
Suppose an application constructs a leap second timestamp (i.e. a struct xtime value with 1000000000<=nsec<2000000000), and passes this timestamp to an xtime primitive that expects a TIME_UTC timestamp.
I don't see why an application would want to construct a bogus leap second. I see no reason to do this One scenario is that the application is importing textual data that ends in :60. The textual data might well be bogus, and the application should be able to check it. This sort of thing is quite common in portable applications that exchange timestamps. If the sec field is 59 mod 60, then there exists at least at least a way to display the leap second, and if the sec field is 83599 mod 84600, then the leap second smells already genuine since it comes at the end of a UTC day. Both tests are trivial. Yes, it's easy to detect _many_ bogus timestamps, but that's not sufficient. You need a way to detect _all_ bogus timestamps. And if the application doesn't have a full leap second table, then you need a way to determine whether the timestamp is known to be bogus, known to be valid, or neither.
Then the behavior is undefined unless the timestamp is not bogus (i.e. it is within a true inserted leap second) _and_ the timestamp is known (i.e. the implementation knows about the leap second). (The current spec doesn't say all this explicitly, but it certainly implies it.)
Just to make sure: We are only talking about xtime_conv and tz_jump here, right? The other functions do not care about bogus leap seconds as long as they come at the end of a minute (sec = 59 (mod 60)). No, I'm talking about all the functions. You write ``the other functions do not care'' because you have a particular implementation in mind that allows bogus leap seconds without checking them. But it's unwise for the spec to _require_ such implementations. The spec should allow higher-quality implementations that do not allow bogus leap seconds at all. Such implementations currently include the Olson code (in "right" mode) and Bernstein's libtai.
So the application needs a way to find out whether a purported leap second is a known leap second....
My suggestion is to use xtime_cmp() in order to define a total order on all know leap seconds and the parameter timestamp, so as to locate the next higher/lower known leap second in a sorted list of leap seconds Sorry, I don't understand this suggestion. Are you proposing that the program initially create a table of known leap seconds by iterating through tz_jump? This sounds awkward, and anyway it won't work reliably in long-running programs, as new leap seconds may become known to the implementation during execution.
One way to fix this particular problem would be to change the spec so that every primitive reports an error when given an bogus or unknown leap second.
No, this is not at all a good idea. I tend to agree; I was proposing it as a possible fix, but suggested a better fix later (the generalized TIME_TAI and modified TIME_UTC discussed below). However, regardless of whether the fix below is adopted, I think the implementation should be allowed to report an error when it detects a bogus or unknown leap second. Just use xtime_cmp() to compare timestamps, and build tz_jump completely on top of xtime_cmp. Sorry, I don't follow this suggestion. You can implement UTC tz_jump by iterating through all potential leap seconds starting from 1972 and ending one year from now; you don't need xtime_cmp. Implementing local-time tz_jump is much harder; in principle you need to iterate through every second (though in practice 8-day iterations suffice). But I don't see why the implementer would build tz_jump that way, unless he was layered above another level that didn't disclose its leap seconds directly. So I don't see where your suggestion is headed.
Suppose implementation X internally uses a clock with TAI seconds, but it doesn't know the TAI epoch;
OK, I assume your "internal clock with TAI seconds" could be one that I could pass directly to the user via TIME_MONOTONIC, Yes. and the epoch could for instance be a "few months ago" (say system installation time). Only if the installation time is at a TAI second boundary, which is unlikely.
First, implementation X can't support struct xtime's TIME_TAI, as this requires knowledge of the TAI epoch.
If we don't know the current TIME_UTC-TIME_TAI and also not the current TIME_MONOTONIC-TIME_TAI, then there is very little we can do to provide TAI at all to the user, Yes you can! You can provide a clock that ticks TAI seconds with an unknown epoch. You can also convert this clock to UTC, so long as the requested times are within the window of known leap seconds. All this is useful information.
This seems backwards to me; fundamentally, the implementation is based on TAI not UTC, and it should have some way to report this to the application.
How can the implementation be based fundamentally on TAI but not know the TAI epoch? Because it gets its timestamps from GPS? Please make sure you have fully understood the difference between TIME_MONOTONIC and TIME_TAI. Both are very similar in that they do not have leap seconds, and the only difference is that TIME_TAI has a known epoch, while TIME_MONOTONIC has not. That's not the only difference! There are two others: (1) TIME_MONOTONIC is monotonically nondecreasing (hence its name), whereas TIME_TAI can go backwards when the clock is reset. (2) TIME_MONOTONIC's seconds may start right in the middle of a TAI second with no problem, whereas TIME_TAI's seconds are supposed to be synchronous with true TAI seconds (perhaps with some small error). I think you are thinking about a (quite reasonable) implementation, that runs its internal clocks on a TIME_MONOTONIC that had its epoch when someone inserted the batteries into the device and that was occationally frequency calibrated when UTC became available over a longer perioid of time. No, I'm assuming something stronger than that for TIME_TAI. I'm assuming not only that it became frequency calibrated, but also that it became ``phase-calibrated'' (sorry, I don't know the lingo, but what I mean is that its second ticks should be very close to the true TAI second ticks), and also that for a certain window around the current time, the relationship between TIME_TAI and UTC is known.
From what you've written, these are both reasonable assumptions for the devices that you're talking about.
Second, implementation X can't convert timestamps outside its window to internal format, so it will reject attempts to invoke primitives involving TIME_UTC timestamps outside its window, as the internal format of those timestamps would be undefined.
Please tell me exactly which operations in my API you think would fail. All operations that take out-of-window struct xtime values would fail, because implementation X converts struct xtime to its internal format (TAI within a window) in order to do all operations. This is a reasonable implementation. Remember that only tz_jump and xtime_conv depend on the leap second history, all other functions do not care about the leap second history This is true of the implementation that you're thinking of, but I don't think the C standard should require such an implementation; it should also allow implementations that are fundamentally TAI based as described above. if you know only all leap seconds since you inserted the batteries and TIME_MONOTONIC started, then xtime_conv will not allow you to convert a TIME_UTC value from before you inserted the batteries to a negative TIME_MONOTONIC value. But why would you want to do this, since you never received a corresponding clock value from the clock anyway? Because you imported a leap-second timestamp from some other source, e.g. from a correspondent over the network using the draft IETF calendar spec.
3. The struct xtime spec does not support applications that do not want to know about leap seconds and do not ever want to see them...
First of all, the existing POSIX functions as well as BSD's gettimeofday are continuing to exist. Not all applications run on POSIX or BSD. And it's messy and error-prone for implementations to combine BSD gettimeofday or POSIX.1-1996 clock_gettime with struct xtime. We should give people a simple way to do common things. You can always call xtime_delay yourself if you are inside a leap second and call xtime_get again afterwards. This is the sort of hack that most users are unlikely to think of on their own -- and also it will impose unacceptable delays on some apps. We need a simple way to get the desired values, without delays. If you want to get a special filter (e.g., the US 59 Hz power grid leap rule or the BSD 10000 ppm adjtimex() rule), then you can implement easily this specific rule using tz_jump (just do: if the last leap second is less than 60 s away, add a linear compensation ramp (delta_t / 60.0) to the time). I thought that tz_jump was supposed to report only discontinuities. Aren't these filters continuous? Now that you mention it, though, it'd be nice if tz_jump (or some new primitive) also reported continuous changes to the TIME_TAI and TIME_UTC clocks as well, if that information is available. This is a good idea, and will better support apps that don't want to know about leap seconds. It is a reasonable extension, though offhand I don't think it needs to be in the standard.
If you combine solutions (2) and (3), then you don't need the current TIME_UTC any more. It's enough to have the generalization of TIME_TAI with implementation-defined epoch, along with the variant of TIME_UTC that never reports leap seconds.
So I can't get the current UTC any more directly from the source? Yes you can. I'm assuming that the actual source is a TAI clock with an unknown epoch, with a known relationship to UTC within a particular window. Within the window, one can say that ``the source'' is TAI or UTC -- it doesn't really matter, and it's a mere notational convenience to say one or the other. Outside the window, you can't convert clock values to broken-down UTC times no matter which notational convention you use. The differences between my proposal and yours are mostly ones of notational convenience; they aren't fundamental to the implementation. My basic argument is that it's much more convenient to base the clock on an integer than to base it on a structure containing a somewhat-broken-down time, with all its foibles. And all conversion functions suddenly depend on the leap second table and on the implementation defined epoch, including all the calculation overhead and reliability concerns involved with this? The (partial) leap second table and implementation-defined epoch is inherent to this sort of implementation. You can't properly support UTC leap seconds without it. In this respect, the ``calculation overhead'' and ``reliability concerns'' of integer-based clocks apply with equal force to struct xtime. Do you really want to send every time stamp on its way from the external UTC reference to the output formatting routine several times through a leap second table Going through a leap second table won't cost much, compared to all the other things that need to be done to convert. This is particularly true for implementations that know at most one leap second -- that table is pretty small. :-) If an app is really concerned about this miniscule overhead, it can avoid the overhead entirely by using TIME_UTC timestamps instead of TIME_TAI timestamps.

Paul Eggert wrote on 1998-10-07 22:37 UTC:
So the application needs a way to find out whether a purported leap second is a known leap second....
My suggestion is to use xtime_cmp() in order to define a total order on all know leap seconds and the parameter timestamp, so as to locate the next higher/lower known leap second in a sorted list of leap seconds
Sorry, I don't understand this suggestion. Are you proposing that the program initially create a table of known leap seconds by iterating through tz_jump?
No.
This sounds awkward, and anyway it won't work reliably in long-running programs, as new leap seconds may become known to the implementation during execution.
Agreed. I was talking about the implementation of tz_jump() using xtime_cmp() internally, because it was my understanding that you thought there might be problems with specifying the precise semantics of tz_time() in the context of bogus leap seconds. My point was that this is trivially resolved by implementing tz_jump internally the following way: Given are to tz_jump() a number of externally known leap seconds (and these might indeed change in long-running implementations), plus an input timestamp. Now you (conceptually) sort all known leap seconds using xtime_cmp(). Then you lookup where the input timestamp would have to be inserted into this sorted list, and you just return the next higher or next lower (depending on the requested direction) known leap second. This is a very clear and straight forward semantic of tz_jump, that gives a well-defined answer even if the input timestamp is not only a bogus leap second, but also out of the valid range. Using tz_jump in order to test whether a given leap second is known or not is also trivial: you set nsec=0 and sec-- in the suspected leap second and give it with direction == +1 to tz_jump. If it returns the same leap second, than the system know about it. If it did not, then either the system has not been informed about the leap second, or what you have is a bogus leap second. We never can distinguish for sure between the two cases anyway, so the API's semantic should depend as little as possible on this distinction.
Just use xtime_cmp() to compare timestamps, and build tz_jump completely on top of xtime_cmp.
Sorry, I don't follow this suggestion.
I hope the above made clear what I meant. (Whether it is the answer to what you asked originally is another question.)
How can the implementation be based fundamentally on TAI but not know the TAI epoch?
Because it gets its timestamps from GPS?
GPS *does* output both TAI and UTC. You are probably more thinking about time services like NTP and DCF77 that only warn about the next leap second but provide no relationship to TAI.
Please make sure you have fully understood the difference between TIME_MONOTONIC and TIME_TAI. Both are very similar in that they do not have leap seconds, and the only difference is that TIME_TAI has a known epoch, while TIME_MONOTONIC has not.
That's not the only difference! There are two others:
(1) TIME_MONOTONIC is monotonically nondecreasing (hence its name), whereas TIME_TAI can go backwards when the clock is reset.
Well, that is just an immediate and direct unavoidable consequence of having a known epoch. If you have a known epoch, then there is the possibility that you learn that you have the wrong epoch and therefore have to correct. If you by definition don't have a known epoch, then it can't be wrong and you will never have to adjust the epoch (or equivalently the pahse of the clock).
(2) TIME_MONOTONIC's seconds may start right in the middle of a TAI second with no problem, whereas TIME_TAI's seconds are supposed to be synchronous with true TAI seconds (perhaps with some small error).
I do not see what advantages it has to have the fourth clock type that you propose, which seems to be identical to TIME_MONOTONIC, except that the difference to TIME_UTC and TIME_TAI is regulated to be an integral number of seconds. What is so special about the second ticks here? If prefer not to have such a rule in TIME_MONOTONIC, because this then allows arbitrary precise and constant frequency of TIME_MONOTONIC, while in your proposal you would have to introduce a phase control loop into TIMe_MONOTONIC. xtime_conv() is perfectly happy if TIME_MONOTONIC-TIME_UTC is 1234567.89 seconds before a leap second and 1234568.89 after a leap second. I see no advantages of keeping the TIME_MONOTONIC-TIME_UTC difference a multiple of integers, while I see big advantages in not requiring this in order to allow highly stable TIME_MONOTONIC implementations.
No, I'm assuming something stronger than that for TIME_TAI. I'm assuming not only that it became frequency calibrated, but also that it became ``phase-calibrated'' (sorry, I don't know the lingo, but what I mean is that its second ticks should be very close to the true TAI second ticks), and also that for a certain window around the current time, the relationship between TIME_TAI and UTC is known.
Better call this PPS-synchronization (PPS = pulse per second, the standard TTL connector output of reference clocks, 500 ms high, 500 ms low, positive edge at the start of the second, to which you can connect a 2-channel 2 GHz digital oscilloscope to match the modulo 1 second phase difference between two reference clocks).
From what you've written, these are both reasonable assumptions for the devices that you're talking about.
The technical reason for why I do not want PPS synchronization in TIME_MONOTONIC is that many types of reference clocks are not immediately available at boot time. A DCF77 receiver needs up to two minutes to lock onto the signal, a GPS receiver needs up to 12 minutes, a NTP server also needs a couple of minutes to make its initial measurements before the PLL stabilizes reasonably well to below millisecond noise, etc. However there are requirements for reliable clocks that are guaranteed to be available right from the first time a C program is started. Leap seconds are not the only uncertainty that we have to deal with, initial or temporary unavailability of clocks, as well as spurious wrong clock readings and fluctuations are another one, and these errors do not come nicely in multiples of one billion nanoseconds. Once we get Caesium clocks for 50 dollars inside a chip package, you will even notice fluctuations in signals from external reference clocks (ionospheric fluctuations for GPS, ground wave versus space wave interference for DCF77, hop fluctuations for WWV) and it will be the task of the Kalman filters in your kernel to compensate these appropriately without affecting the stability of your own monotonic reference timescale. PPS-synchronization seems to be a crude and bizarre requirement in this context. But if you really need it: You can easily construct the time you are talking about in your application the following way: 1) As long as TIME_UTC is not yet available, you just use TIME_MONOTONIC. 2) Once UTC has become available, you determine x = TIME_MONOTONIC - TIME_UTC one single time (let's call this moment your post-epoch), and then remove the non-fractional part of x (x.sec = 0). 3) From then on, you use TIME_MONOTONIC - x instead of TIME_MONOTONIC, if there is really a need for a monotonic time with PPS synchronization. You can apply a lowpass filter to smooth the transition from 0 to x if you want. You will have to live with that there is an x-wide gap between the time before and after your post-epoch, unless you go at the post-epoch over all your old time-stamps and convert them by subtracting x to the new PPS-synchronized TIME_MONOTONIC. (If you do this, you could also use x = TIME_MONOTONIC - TIME_TAI without truncation once TAI is available, and then replace all older timestamps by proper TAI timestamps. You could also use TIME_UTC at the time of the post-epoch instead. There are many things you can do here, depending on your needs and personal preferences) As you see, it is easy to implement your clock API on top of mine (but not vice versa). However, if you ever run into an application, where you need this PPS-synchronized monotonic time, please let me know. I certainly cannot think of any right now, therefore I don't want to add another clock type to my proposal that can easily be derived by the applications in numerous ways from the existing one. It is important however to understand that and how even such more exotic clock types can be easily emulated on top of it.
All operations that take out-of-window struct xtime values would fail, because implementation X converts struct xtime to its internal format (TAI within a window) in order to do all operations. This is a reasonable implementation.
No. My API is only reasonably implemented by operating *directly* on struct xtime values. If you think that it is reasonable to implement anything on some other internal format, then you have probably misunderstood something. The actual clock registers that hold the current time are the only places where you might have any of a large number of implementation-dependent representations (often dictated by the existing OS kernel or by the hardware clock format).
Remember that only tz_jump and xtime_conv depend on the leap second history, all other functions do not care about the leap second history
This is true of the implementation that you're thinking of, but I don't think the C standard should require such an implementation; it should also allow implementations that are fundamentally TAI based as described above.
xtime_get() can use whatever it wants internally, but for all other implementations, there is little need to introduce an extra internal time representation beyond something very closely related to xtime itself.
if you know only all leap seconds since you inserted the batteries and TIME_MONOTONIC started, then xtime_conv will not allow you to convert a TIME_UTC value from before you inserted the batteries to a negative TIME_MONOTONIC value. But why would you want to do this, since you never received a corresponding clock value from the clock anyway?
Because you imported a leap-second timestamp from some other source, e.g. from a correspondent over the network using the draft IETF calendar spec.
And, what harm can it do to you? I have shown you functions that you can use to savely use this other timestamp in calculations with well-defined semantics, even if they are bogus, and if you try to convert it using xtime_conv() to TIME_TAI or TIME_MONOTONIC, then the implementation will certainly tell you whether this is possible. Doing this is however not advisable. It is in most cases preferable if you go the other way round and convert you own time_stamps to TIME_UTC before doing any comparisons, etc. I think I have now completely understood what your API model is and what you are thinking about it in relation to xtime. I also think that you are on the wrong path: You are analysing in some sense xtime by trying to implement it on top of something similar to libtai and you discover that this is not possible conveniently. Instead of recognizing that the real problem is that a libtai-style lower layer has a much more restricted functionality and that therefore this type of implementation is not a good idea for a universal standard clock API (although it might be highly useful for certain types of special application, such as accounting which Dan Bernstein mentioned), you try to redesign and restrict my API to make it implementable on top of a restricted pseudo-TAI like clock model. Why do you think this is a good thing?
You can always call xtime_delay yourself if you are inside a leap second and call xtime_get again afterwards.
This is the sort of hack that most users are unlikely to think of on their own -- and also it will impose unacceptable delays on some apps. We need a simple way to get the desired values, without delays.
Yes, but we do not know WHAT the desired values are!
If you want to get a special filter (e.g., the US 59 Hz power grid leap rule or the BSD 10000 ppm adjtimex() rule), then you can implement easily this specific rule using tz_jump (just do: if the last leap second is less than 60 s away, add a linear compensation ramp (delta_t / 60.0) to the time).
I thought that tz_jump was supposed to report only discontinuities. Aren't these filters continuous?
Tz_jump() reports only discontinuities. Think of the leap second discontinuity as a dirac pulse, which you convolute with your filter function, and the resulting sequences of soft falling steps is what you add to your leap-filtered time. without system-theory lingo: You use tz_jump only to measure how far away the last leap second is when you read a clock, and then you add the height of the compensation ramp that is adequate for the reported distance to the leap second. You will normally use finite-impulse response filters, i.e. the frequency of the filtered clock is correct again after say 1 or 20 minutes.
Now that you mention it, though, it'd be nice if tz_jump (or some new primitive) also reported continuous changes to the TIME_TAI and TIME_UTC clocks as well, if that information is available.
Instead of reporting drifts between TIME_UTC | TIME_SYNC and TIME_MONOTONIC, you should steer the frequency of TIME_MONOTONIC to be identical to that of TIME_UTC | TIME_SYNC. Then the API does not provide you with any continuous changes that are worth reporting.
This is a good idea, and will better support apps that don't want to know about leap seconds. It is a reasonable extension, though offhand I don't think it needs to be in the standard.
If you want, you can easily add your own xtime_get_smooth_UTC() function that takes as argument a maximum frequency deviation (the steepness of the ramp), and gives you pack a leap-second free UTC that is free of all discontinuities reported by tz_jump. You can filter using linear ramps, Gaussian bells, Bezier splines, and whatever you want. If there is a very wide need, then I can provide a sample implementation for a linear ramp filter with selectable frequency deviation. The important thing is that xtime_get and tz_jump are all that is needed to implement these, i.e. the right infrastructure is in place.
My basic argument is that it's much more convenient to base the clock on an integer than to base it on a structure containing a somewhat-broken-down time, with all its foibles.
My basic argument is that if you use just an integer, most algorithms will get more complicated if you provide for leap seconds, while the algorithms continue to look pretty much the same as what we have for POSIX systems if you use xtime.
The (partial) leap second table and implementation-defined epoch is inherent to this sort of implementation. You can't properly support UTC leap seconds without it. In this respect, the ``calculation overhead'' and ``reliability concerns'' of integer-based clocks apply with equal force to struct xtime.
In your implementation, you need a (partial) leap second table each time you want to convert to a broken down time, while in my implementation you need it only to do conversion between clock types (including filtered ones). If you always use real UTC (and I think that is what most applications will want to use, because that is what legal local time is based on), you never use a leap second table (except for the internal kernel warning flag that one is coming at the end of the next minute). Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
you need a (partial) leap second table each time you want to convert to a broken down time,
Leap seconds have been part of the definition of local time since 1972. Why do you find them more frightening than Daylight Savings Time and the Gregorian calendar? ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

Date: Fri, 09 Oct 1998 12:17:31 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk>
(1) TIME_MONOTONIC is monotonically nondecreasing (hence its name), whereas TIME_TAI can go backwards when the clock is reset.
Well, that is just an immediate and direct unavoidable consequence of having a known epoch. True, but it's an important consequence that is not immediately obvious to most people, so when describing the difference between TIME_MONOTONIC and TIME_TAI it's definitely worth mentioning.
(2) TIME_MONOTONIC's seconds may start right in the middle of a TAI second....
I do not see what advantages it has to have the fourth clock type that you propose, which seems to be identical to TIME_MONOTONIC, except that the difference to TIME_UTC and TIME_TAI is regulated to be an integral number of seconds. What is so special about the second ticks here? Because this fourth clock type (which I am calling the ``modified TIME_TAI'') can be implemented easily from NTP and DCF77 without complete leap second tables; whereas the original struct xtime TIME_TAI cannot. And once we have the modified TIME_TAI, we don't need nsec>=1000000000, and this cleans up several infelicities in the current interface. If prefer not to have such a rule in TIME_MONOTONIC, because this then allows arbitrary precise and constant frequency of TIME_MONOTONIC, while in your proposal you would have to introduce a phase control loop into TIMe_MONOTONIC. Sorry, I don't understand this comment. I wasn't proposing to change TIME_MONOTONIC; I was proposing to change TIME_TAI. TIME_MONOTONIC can stay as it is. xtime_conv() is perfectly happy if TIME_MONOTONIC-TIME_UTC is 1234567.89 seconds before a leap second and 1234568.89 after a leap second. Agreed. Let's leave TIME_MONOTONIC alone. You can easily construct the time you are talking about in your application the following way: Yes, one can construct modified TIME_TAI, given the original TIME_TAI. But my point is that the modified TIME_TAI is easier to implement than the original TIME_TAI, and it has more useful properties in practice (in particular, it means we don't need nsec>=1000000000). Another way to put it is that the modified TIME_TAI is a generalization of the original TIME_TAI, one that places less of a burden on implementations than the original TIME_TAI proposal. It allows high-quality implementations with full leap second tables, but it also allows lower-quality implementations that have only partial tables. if you ever run into an application, where you need this PPS-synchronized monotonic time, please let me know. The application I'm thinking of is one where the original TIME_TAI is not available, but the modified TIME_TAI is available, and we want to support leap seconds within the window supported by the partial leap second table. From what you've written earlier, this is common with implementations that get times from NTP and DCF77 but do not have complete leap second tables. you try to redesign and restrict my API to make it implementable on top of a restricted pseudo-TAI like clock model. Why do you think this is a good thing? Because the C standard ought to be broadly applicable. It ought to allow high-quality implementations like Bernstein's or Olson's "right" mode, as well as lower-quality implementations like POSIX.1, and also medium-quality implementations like NTP without full leap second tables. The standard should not require high-quality implementations, as that's unrealistic; but also, it should not require low-quality implementations, as they don't suffice for some applications. My basic argument is that if you use just an integer, most algorithms will get more complicated if you provide for leap seconds, I disagree. Let's try to resolve this argument by doing things both ways, and seeing which works better. I'll write up a spec that uses integers, and which has the functionality of struct xtime, and then we can see which is easier to use on some realistic examples. Using tz_jump in order to test whether a given leap second is known or not is also trivial: you set nsec=0 and sec-- in the suspected leap second It's a bit more complicated than that, since you can't always subtract 1 from sec, as this might overflow. Of course this can be worked around, but the interface is trickier than it could be. If it returns the same leap second, than the system know about it. If it did not, then either the system has not been informed about the leap second, or what you have is a bogus leap second. We never can distinguish for sure between the two cases anyway Yes we can! For example, the implementation can know that there is no leap second _tonight_. There ought to be some way that the programmer can get this information, if available.
How can the implementation be based fundamentally on TAI but not know the TAI epoch?
Because it gets its timestamps from GPS?
GPS *does* output both TAI and UTC. You are probably more thinking about time services like NTP and DCF77 Yes, that's what I was thinking about. Sorry about the confusion.

Paul Eggert writes:
which I am calling the ``modified TIME_TAI''
Your time differences break down near leap seconds. Ever watched xntpd try to handle a leap second? Wobble, wobble, wobble, wobble, wobble. After that, if you don't record the leap second, you'll start printing incorrect local times from before the leap second, for the same reason that Markus starts printing incorrect time differences. The solution is obvious: keep a table of leap seconds! There's no other way to handle UTC timestamps correctly. ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

Date: 9 Oct 1998 23:40:07 -0000 From: "D. J. Bernstein" <djb@cr.yp.to> Paul Eggert writes:
which I am calling the ``modified TIME_TAI''
Your time differences break down near leap seconds. Ever watched xntpd try to handle a leap second? Wobble, wobble, wobble, wobble, wobble. Sorry, I don't understand this comment. The modified TIME_TAI has an implementation-defined epoch, but the epoch is supposed to be fixed for the duration of the execution of the program. If TIME_TAI is implemented accurately, then time differences will be accurate in all timestamps covered by the (partial) leap second table available to the implementation. If the underlying implementation is flaky, then obviously the TIME_TAI values will be flaky; but this is a quality-of-implementation issue, not a fundamental flaw in the interface. After that, if you don't record the leap second, Under my proposal, you'd have to record the leap second, if you wanted to implement modified TIME_TAI correctly, and if you had already handed out some modified TIME_TAI timestamps. The solution is obvious: keep a table of leap seconds! There's no other way to handle UTC timestamps correctly. I tend to agree. My proposal generalizes from libtai in that it allows leap second tables that do not go all the way back to 1972, in an attempt to support the sorts of limited implementations that Kuhn mentions; but I see no way around having a table (if only a partial one) if you want to support leap seconds correctly (if only some of them).

If you use libtai, but you fail to set up a leap-second table, you get POSIX time_t: the number of non-leap seconds since the epoch. Paul Eggert writes:
Under my proposal, you'd have to record the leap second,
Okay. In that case you're simply talking about what libtai does when your leap-second table doesn't go back to 1972. ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> Resent-From: tz@elsie.nci.nih.gov The only doubts that I have about xtime is that I think it is with 96 bits a bit too long. I would feel slightly better with 64 bits. Yes, yes! This is the Right Way to Go. And we can do it, while still satisfying your other concerns. It would also completely mess up the elegance of my API in that suddenly strfxtime and xtime_make/breakup would have to know from what type of clock this timestamp came. strfxtime and the other primitives currently return garbage unless you pass them a TIME_UTC timestamp. I suppose one could argue that they return well-defined garbage; but this well-defined garbage is useless for practical applications. This is a defect in the current proposal, and I plan to propose a fix soon. But no matter how it's fixed, it's inevitable that either (A) time-zone-related primitives will have to know the timestamp's clock type, or (B) such primitives will be useful only on TIME_UTC timestamps. (A) is better than (B), particularly if we adopt solutions (2) and (3) mentioned below.
So I suggest that we give the programmer a way to access to the entire leap second table known to the implementation.
My API does this in form of tz_jump. Yes, but I see that I wasn't clear. Let me try to explain things better. I see three related problems with struct xtime in this area. 1. The struct xtime spec makes it difficult for applications to detect bogus or unknown leap seconds. Suppose an application constructs a leap second timestamp (i.e. a struct xtime value with 1000000000<=nsec<2000000000), and passes this timestamp to an xtime primitive that expects a TIME_UTC timestamp. Then the behavior is undefined unless the timestamp is not bogus (i.e. it is within a true inserted leap second) _and_ the timestamp is known (i.e. the implementation knows about the leap second). (The current spec doesn't say all this explicitly, but it certainly implies it.) Therefore, when importing a textual leap-second timestamp from the outside world, a struct xtime application can't blindly apply xtime_make and use the resulting value, as xtime_make might generate a bogus or unknown leap-second timestamp, which will cause problems when given to other primitives. So the application needs a way to find out whether a purported leap second is a known leap second. It can't pass the purported leap second directly to tz_jump, because if it's a bogus leap second tz_jump has undefined behavior. I can think of hacky workarounds to this problem, but this is getting way too complicated; there ought to be a simpler way to attack this problem. One way to fix this particular problem would be to change the spec so that every primitive reports an error when given an bogus or unknown leap second. This is an improvement, but it's not enough to address the other problems mentioned below; and solving the other problems will solve this problem. 2. The struct xtime spec is an awkward interface to implementations based on TAI that do not have full leap second tables. Suppose implementation X internally uses a clock with TAI seconds, but it doesn't know the TAI epoch; that is, it knows the leap second or seconds within a window of a few months around the current time, and it knows the relationship between its internal clock and UTC within this window, but it doesn't know all the leap seconds back to 1972. This seems to be the kind of implementation that is motivating your leap-second design, but I see a couple of problems with how struct xtime would interface to implementation X. First, implementation X can't support struct xtime's TIME_TAI, as this requires knowledge of the TAI epoch. This seems backwards to me; fundamentally, the implementation is based on TAI not UTC, and it should have some way to report this to the application. Second, implementation X can't convert timestamps outside its window to internal format, so it will reject attempts to invoke primitives involving TIME_UTC timestamps outside its window, as the internal format of those timestamps would be undefined. This seems counterintuitive to me. From the user's point of view, we'd have a system that claims to support TIME_UTC but can't convert the out-of-window value 1970-01-01 00:00:00 UTC to a TIME_UTC timestamp, even though the result is perfectly well-defined to be zero (the problem being that the result's _internal form_ is unknown). A more natural way to support such an implementation is to have the struct xtime interface supply a generalization of TIME_TAI, one with an implementation-defined epoch. This generalization would provide a more natural interface to this class of implementations, and would also support systems with complete knowledge of past leap seconds. 3. The struct xtime spec does not support applications that do not want to know about leap seconds and do not ever want to see them. This is a large class of applications, and POSIX.1 (as well as many important non-POSIX OSes) support such applications, but the struct xtime spec doesn't support them at all. This can be fixed by adding a variant of TIME_UTC that never reports leap seconds. If you combine solutions (2) and (3), then you don't need the current TIME_UTC any more. It's enough to have the generalization of TIME_TAI with implementation-defined epoch, along with the variant of TIME_UTC that never reports leap seconds. Also, once you combine solutions (2) and (3), you don't ever need struct xtime values with nsec>=1000000000. If an application wants to work properly with leap seconds, it should use the generalization of TIME_TAI and thus doesn't need nsec>=1000000000. Conversely, if the application doesn't want to know about leap seconds and doesn't ever want to see them, it should use the variant of TIME_UTC that never reports leap seconds, so it also doesn't need nsec>=1000000000. Disallowing nsec>=1000000000 simplifies things greatly. It also makes it practical to use a single 64-bit integer timestamp instead of a two-part structure. I'm working on a proposal along these lines (derived from your proposal) and would like to publish it soon.

Paul Eggert wrote on 1998-10-07 08:45 UTC:
It would also completely mess up the elegance of my API in that suddenly strfxtime and xtime_make/breakup would have to know from what type of clock this timestamp came.
strfxtime and the other primitives currently return garbage unless you pass them a TIME_UTC timestamp. I suppose one could argue that they return well-defined garbage; but this well-defined garbage is useless for practical applications.
No, they return useful values if you pass them TIME_TAI or TIME_LOCAL (both of which have a 1970 epoch as well), and specify (timezone_t * NULL) as the other parameter. Only if you pass a TIME_MONOTONIC value, you might get real garbage (except on those systems where TIME_MONOTONIC is a best effort TAI approximation, which is certainly allowed though not required). See my last message for more details.
This is a defect in the current proposal, and I plan to propose a fix soon.
No, I think you misunderstood the proposal here. What you inappropriately call "well-defined garbage" is fully intended to be useful for TIME_TAI or TIME_LOCAL.
Yes, but I see that I wasn't clear. Let me try to explain things better. I see three related problems with struct xtime in this area.
1. The struct xtime spec makes it difficult for applications to detect bogus or unknown leap seconds.
Suppose an application constructs a leap second timestamp (i.e. a struct xtime value with 1000000000<=nsec<2000000000), and passes this timestamp to an xtime primitive that expects a TIME_UTC timestamp.
I don't see why an application would want to construct a bogus leap second. I see no reason to do this and I feel comfortable leaving the behaviour undefined here (although I do propose a clear and safe semantic below using xtime_cmp()). Creating bogus leap seconds is like dividing by zero: Whatever will happen is your fault anyway. Garbage in, garbage out. If the sec field is 59 mod 60, then there exists at least at least a way to display the leap second, and if the sec field is 83599 mod 84600, then the leap second smells already genuine since it comes at the end of a UTC day. Both tests are trivial.
Then the behavior is undefined unless the timestamp is not bogus (i.e. it is within a true inserted leap second) _and_ the timestamp is known (i.e. the implementation knows about the leap second). (The current spec doesn't say all this explicitly, but it certainly implies it.)
Just to make sure: We are only talking about xtime_conv and tz_jump here, right? The other functions do not care about bogus leap seconds as long as they come at the end of a minute (sec = 59 (mod 60)).
Therefore, when importing a textual leap-second timestamp from the outside world, a struct xtime application can't blindly apply xtime_make and use the resulting value, as xtime_make might generate a bogus or unknown leap-second timestamp, which will cause problems when given to other primitives.
No, a textual representation only can create a sec = 59 (mod 60) bogus leap second, and they can be adequately displayed, so no information will be lost.
So the application needs a way to find out whether a purported leap second is a known leap second. It can't pass the purported leap second directly to tz_jump, because if it's a bogus leap second tz_jump has undefined behavior. I can think of hacky workarounds to this problem, but this is getting way too complicated; there ought to be a simpler way to attack this problem.
If you are concerned about the precise definition of tz_jump() for pathological input: My suggestion is to use xtime_cmp() in order to define a total order on all know leap seconds and the parameter timestamp, so as to locate the next higher/lower known leap second in a sorted list of leap seconds, and voila, you immediately have a clean semantic of where the next one is, even in the context of completely arbitrary xtime values (not only bogus leap seconds, but also for out of range nsec values). Would this be acceptable for you?
One way to fix this particular problem would be to change the spec so that every primitive reports an error when given an bogus or unknown leap second.
No, this is not at all a good idea. It adds a lot of unnecessary check mechanics, and it messes up the operation on systems that are informed by external clocks on the fly that there is now a leap second coming up, but then then forget quickly about leap seconds and do not store any leap seconds (because they are for instance embedded controllers with no non-volatile memory). Just use xtime_cmp() to compare timestamps, and build tz_jump completely on top of xtime_cmp.
2. The struct xtime spec is an awkward interface to implementations based on TAI that do not have full leap second tables.
Suppose implementation X internally uses a clock with TAI seconds, but it doesn't know the TAI epoch;
OK, I assume your "internal clock with TAI seconds" could be one that I could pass directly to the user via TIME_MONOTONIC, and the epoch could for instance be a "few months ago" (say system installation time).
that is, it knows the leap second or seconds within a window of a few months around the current time, and it knows the relationship between its internal clock and UTC within this window, but it doesn't know all the leap seconds back to 1972. This seems to be the kind of implementation that is motivating your leap-second design, but I see a couple of problems with how struct xtime would interface to implementation X.
First, implementation X can't support struct xtime's TIME_TAI, as this requires knowledge of the TAI epoch.
If we don't know the current TIME_UTC-TIME_TAI and also not the current TIME_MONOTONIC-TIME_TAI, then there is very little we can do to provide TAI at all to the user, since we obviously just do not have it available.
This seems backwards to me; fundamentally, the implementation is based on TAI not UTC, and it should have some way to report this to the application.
How can the implementation be cased fundamentally on TAI but not know the TAI epoch? Please make sure you have fully understood the difference between TIME_MONOTONIC and TIME_TAI. Both are very similar in that they do not have leap seconds, and the only difference is that TIME_TAI has a known epoch, while TIME_MONOTONIC has not. I think you are thinking about a (quite reasonable) implementation, that runs its internal clocks on a TIME_MONOTONIC that had its epoch when someone inserted the batteries into the device and that was occationally frequency calibrated when UTC became available over a longer perioid of time. I think I have carefully thought about the application scenario you refer to, and you just haven't understood yet that what you really want here is TIME_MONOTONIC (the TIME_TAI without known epoch) and not TIME_TAI.
Second, implementation X can't convert timestamps outside its window to internal format, so it will reject attempts to invoke primitives involving TIME_UTC timestamps outside its window, as the internal format of those timestamps would be undefined.
Please tell me exactly which operations in my API you think would fail. Remember that only tz_jump and xtime_conv depend on the leap second history, all other functions do not care about the leap second history and will never fail because of a lack of leap second information. Sure, if you know only all leap seconds since you inserted the batteries and TIME_MONOTONIC started, then xtime_conv will not allow you to convert a TIME_UTC value from before you inserted the batteries to a negative TIME_MONOTONIC value. But why would you want to do this, since you never received a corresponding clock value from the clock anyway? And as far as calendar and timezone calculations are concerned: just stay in TIME_UTC. There is no reason to ever convert to your internal TIME_MONOTONIC timescale as long as your internal clocks are not involved in any way in the calculations. If your internal clocks get involved, use xtime_conv() to convert from TIME_MONOTONIC to TIME_UTC and everything is alright again. I don't think you have found a problem. Actually, the discussion just convinces me more and more about the flexibility and completeness of the xtime API.
This seems counterintuitive to me. From the user's point of view, we'd have a system that claims to support TIME_UTC but can't convert the out-of-window value 1970-01-01 00:00:00 UTC to a TIME_UTC timestamp, even though the result is perfectly well-defined to be zero (the problem being that the result's _internal form_ is unknown).
Please be more precise and explain, which function you think would fail.
A more natural way to support such an implementation is to have the struct xtime interface supply a generalization of TIME_TAI, one with an implementation-defined epoch.
That is why TIME_MONOTONIC does exist. But for UTC related display, it becomes only useful once the TIME_MONOTONIC to TIME_UTC conversion has become possible.
This generalization would provide a more natural interface to this class of implementations, and would also support systems with complete knowledge of past leap seconds.
3. The struct xtime spec does not support applications that do not want to know about leap seconds and do not ever want to see them. This is a large class of applications, and POSIX.1 (as well as many important non-POSIX OSes) support such applications, but the struct xtime spec doesn't support them at all.
This can be fixed by adding a variant of TIME_UTC that never reports leap seconds.
First of all, the existing POSIX functions as well as BSD's gettimeofday are continuing to exist. There are several methods of handling leap seconds in them. One is to block until the leap second is over, the other is to send the time through a low-pass filter, i.e. to slow down the clock by say 1% until UTC has caught up with this leap-second filtered time. (The latter is exactly clocks that get their time base from the power grid do.) I am not sure we should offer this in a new interface, especially since there are so many ways of doing this. You can always call xtime_delay yourself if you are inside a leap second and call xtime_get again afterwards. Then you know at least precisely what you get and do not depend on probably not fully specified filtering algorithms. If you want to get a special filter (e.g., the US 59 Hz power grid leap rule or the BSD 10000 ppm adjtimex() rule), then you can implement easily this specific rule using tz_jump (just do: if the last leap second is less than 60 s away, add a linear compensation ramp (delta_t / 60.0) to the time).
If you combine solutions (2) and (3), then you don't need the current TIME_UTC any more. It's enough to have the generalization of TIME_TAI with implementation-defined epoch, along with the variant of TIME_UTC that never reports leap seconds.
So I can't get the current UTC any more directly from the source? And all conversion functions suddenly depend on the leap second table and on the implementation defined epoch, including all the calculation overhead and reliability concerns involved with this? I have yet to read your final proposal to make a judgement, but this sounds not very promising and seems to reintroduce all the problems that I was so proud of having gotten rid of (if it is the completely leap-second table state dependent system that I fear it is). Do you really want to send every time stamp on its way from the external UTC reference to the output formatting routine several times through a leap second table just to make the SI second difftime implementation equivalent to a subtraction? Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Paul Eggert wrote:
strfxtime and the other primitives currently return garbage unless you pass them a TIME_UTC timestamp. I suppose one could argue that they return well-defined garbage; but this well-defined garbage is useless for practical applications.
How about encoding the time type in the struct xtime itself? That permits safe error checking and the use of mixed time types particularly UTC vs. local in contexts like logs -- some devices that need to make log entries may not know the UTC time, only some random local time.
Suppose implementation X internally uses a clock with TAI seconds, but it doesn't know the TAI epoch;
Are not TAI seconds precisely the same as UTC seconds? TAI minutes, hours, days, months, years are different, but not seconds, IIRC: 1 TAI second = 1 UTC second = 9,192,631,770 cesium vibrations.
that is, it knows the leap second or seconds within a window of a few months around the current time, and it knows the relationship between its internal clock and UTC within this window, but it doesn't know all the leap seconds back to 1972.
I think that a more likely state is that it knows the earlier leap seconds between Now and 1972, but is missing some number (potentially zero) of the later ones. This corresponds to an un-updated leap second table.
This seems backwards to me; fundamentally, the implementation is based on TAI not UTC, and it should have some way to report this to the application.
What does it mean to be "based on TAI" at the second level?
If you combine solutions (2) and (3), then you don't need the current TIME_UTC any more. It's enough to have the generalization of TIME_TAI with implementation-defined epoch, along with the variant of TIME_UTC that never reports leap seconds.
A TIME_UTC that doesn't report leap seconds is not UTC at all, but TAI-10. Its minutes are always exactly 60 sec long. -- John Cowan http://www.ccil.org/~cowan cowan@ccil.org You tollerday donsk? N. You tolkatiff scowegian? Nn. You spigotty anglease? Nnn. You phonio saxo? Nnnn. Clear all so! 'Tis a Jute.... (Finnegans Wake 16.5)

Markus Kuhn writes:
The best atomic clock on this planet (CS1 by PTB in Braunschweig) barely can do UTC with a real-time precision of one nanosecond.
There are many scientific projects for which nanosecond resolution is inadequate. ``It's the intervals, stupid.''
I think attoseconds are horrible overkill.
What exactly do you claim is so ``horrible''? Do you think that you're going to run out of disk space if you store 16-byte timestamps? ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

"D. J. Bernstein" wrote on 1998-10-03 04:39 UTC:
Markus Kuhn writes:
The best atomic clock on this planet (CS1 by PTB in Braunschweig) barely can do UTC with a real-time precision of one nanosecond.
There are many scientific projects for which nanosecond resolution is inadequate. ``It's the intervals, stupid.''
I don't see, how the standard C time API would be related to any of these applications. You would use in these scientific applications special purpose instrumentation with special purpose control software that does not even touch the C time API. We are not trying to solve any problem related in any way to any kind of time, we are just trying to define a robust interface to the clock information available to a good operating system on a wide range of general-purpose computers. Tell me some real-world examples where subnanoseconds play a role in applications running on general-purpose computers and where access to operating system clocks and timezone databases is required with subnanosecond resolution to convince me instead of calling me ``stupid''. I am somewhat familiar with radio-astronomy instrumentation and requirements for VLBI, but I have extreme difficulties to imagine that this subnanosecond data would be processed through the general-purpose C API and I see absolutely no need to extend it. I am convinced that nanosecond resolution provides a very comfortable safety margin to the resolution actually required by C applications from the standard library. You certainly do not measure the position of your radio telescope with 3 cm precision (0.1 ns) by performing time request calls to your C API! You would use external picosecond hardware phase comparators clocked by caesium time normals, which send you the data as ASCII floating point numbers for further processing in your Fortran tools (all available from Hewlett Packard for a few kilodollars for the advanced hobby radio astronomer). Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
we are just trying to define a robust interface to the clock information available to a good operating system
Most modern processors provide exact cycle counts. A good clock routine can combine cycle-count information with occasional TAI/UTC inputs to provide reasonably accurate local-time displays with extremely accurate time differences. The available resolution is going to drop below 1ns in a few years, so limiting yourself to 1ns seems rather shortsighted. (Yes, Virginia, you really can tell the difference on a Pentium 233 between something that takes 130ns and something that takes 135ns.)
external picosecond hardware phase comparators clocked by caesium time normals, which send you the data as ASCII floating point numbers
Why not settle on a 16-byte format that works for everything? If you're desperate to save space, you can truncate TAI64NA at the precision you need---8 bytes for TAI64 or 12 bytes for TAI64N. ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

Date: Sun, 04 Oct 1998 12:03:49 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> You certainly do not measure the position of your radio telescope with 3 cm precision (0.1 ns) by performing time request calls to your C API! You would use external picosecond hardware phase comparators clocked by caesium time normals, which send you the data as ASCII floating point numbers for further processing in your Fortran tools It sounds like you're saying that C should not support high-resolution timestamps, and that people who need such things should use Fortran instead. Perhaps I'm paraphrasing you incorrectly; still, I am extremely uncomfortable with the idea that the C standard should impose an upper limit on the quality of the clock. It's reasonable for the C Standard to impose a _lower_ limit on clock quality, just as it imposes a lower limit on the size of `int'. But it's not reasonable for the standard to impose an upper limit, unless there is a compelling justification, which is absent here. The standard places no upper bound on the size of `int', and this is for a good reason: we don't want to forbid higher-quality implementations. I can't offhand recall any place in the existing standard that imposes an upper bound on quality; I don't think <time.h> should either.

Paul Eggert wrote on 1998-10-05 20:44 UTC:
Date: Sun, 04 Oct 1998 12:03:49 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk>
You certainly do not measure the position of your radio telescope with 3 cm precision (0.1 ns) by performing time request calls to your C API! You would use external picosecond hardware phase comparators clocked by caesium time normals, which send you the data as ASCII floating point numbers for further processing in your Fortran tools
It sounds like you're saying that C should not support high-resolution timestamps, and that people who need such things should use Fortran instead.
No, of course not. Please use a bit more common sense (this also addresses some comments made by Dan Bernstein about the need for attoseconds): Fortran does not have a higher resolution time API. For scientific applications, you just use whatever representation of numbers with the required precision your language offers. For these applications, you *anyway* must use handwritten inline assembler timing code in order to access the hardware counter in a strictly deterministic way, taking care of interrupts, caches, TLBs, pipelines, and all the other real-world timing hazards that are completely outside the scope of the C standard and what xtime_get() is intended for! In DSP applications, you do not even convert to seconds but calculate in a clock-dependent time scale and just adjust your constants. DSP applications are also far outside the scope of <time.h>! If you think that an API that does not provide this low-level control over timing hazards can be useful for sub-nanoseconds measurements, then this just indicates that you probably never have tried to use computers yourself for high-precision timing with a 10 GHz oscilloscope next to the machine to verify the results. I would never use a vendor provided compiler library for such applications (and I *have* spent days debugging C compiler timing functions under MS-DOS with oscilloscopes for sub-millisecond hard real-time applications). Of course you can time the position of your radio telescope with nanometer precision in C. There is no reason why you would want to use xtime to represent the quantities in the calculations involved in doing this. You will just use normal floating point variables in whatever time scale your instruments and your application define. The timing variances of your processor (cache flushes, TBL reloads, multi-tasking, bus arbitration, etc.) are at least two orders of magnitude worse than 1 ns, therefore, you will never want to use anything like xtime_get() itself as a source of sub-nanosecond timing information. You will always need special hardware or assembler-level driver design for these things, and if you think that the Pentium's internal bus cycle counter is suitable for such applications, then I suggest you try building a GPS software receiver (100 ns resolution required) on top of it to see the real-life limits of software timing.
Perhaps I'm paraphrasing you incorrectly; still, I am extremely uncomfortable with the idea that the C standard should impose an upper limit on the quality of the clock.
I am not. The limit of 1 ns is *much* smaller (at least two orders of magnitude) than any realistic requirement for this type of a software clock. We are moving *very* far away from the real problems, towards topics that are best described as "academic" in the most negative sense of the word. I hope this does not give some observers the impression that the proposal does indeed need as much discussion as it does get here. I am starting to get concerned that I might have wasted a lot of time while working on this, especially since I didn't get any feedback yet from the only ISO C committee member on this list. To rephrase from a Tony Blair speech a few days ago: Please keep in mind that you do not have the choice between the xtime you have got and the xtime you want. You have the choice between xtime and tmx. :-) Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn writes:
and if you think that the Pentium's internal bus cycle counter
Do you have any idea what you're talking about? The bus frequency is much lower than the CPU frequency.
is suitable for such applications,
As I said, you can tell the difference on a Pentium-233 between something that takes 130ns and something that takes 135ns. Why do you have so much trouble grasping the concept that small intervals have much smaller errors than large intervals? ---Dan 1000 recipients, 28.8 modem, 10 seconds. http://pobox.com/~djb/qmail/mini.html

On Tue, 6 Oct 1998, Markus Kuhn wrote:
I am starting to get concerned that I might have wasted a lot of time while working on this, especially since I didn't get any feedback yet from the only ISO C committee member on this list.
To rephrase from a Tony Blair speech a few days ago:
Please keep in mind that you do not have the choice between the xtime you have got and the xtime you want. You have the choice between xtime and tmx.
:-)
To quote from this same list's archives, in 1987 (views may not reflect the now current views of the then participants): Date: Tue, 10 Mar 87 10:00:19 cst From: seismo!cuuxb!dlm (Dennis L. Mumaugh) Message-Id: <8703101600.AA10580@cuuxb> To: elsie!tz Subject: List structure and standards making While the list has been discussing time and timezones in context of the IEEE P1003 standards committee, people have lost sight of X3J11 which is much more important and further along towards casting into concrete. (Dramatic effect) As we now talk, the concrete has been laid and is hardening. (End drama). People should get out their X3J11 Draft Proposed American Standard for Information Systems -- Programming Language C (Dated October 1, 1986). This standard is ALREADY entering the balloting process. Please turn to page 151 and following: Section 4.12 -- Date and Time. The standard ALREADY covers clock_t, time_t, struct tm (without new strings or fields!!), clock(), time(), difftime(), mktime(), asctime(), ctime(), gmtime(), localtime(), strftime(). Thus the current discussions on the whole time subject are rapidly becomming academic as the ANSI C is already spoken and the C standard is currently in the comment phase -- send a comment and proposed change of ruling. The next phase is balloting and trial use. [snip] From: seismo!munnari!basser.oz!john Message-Id: <8703121839.AA07624@seismo.CSS.GOV> Date: Fri, 13 Mar 87 04:54:50 EST To: cbosgd!cuuxb!dlm Cc: elsie!tz Subject: Re: List structure and standards making > Thus the current discussions on the whole time subject are > rapidly becomming academic Dennis, The discussions are certainly NOT becoming academic. There are a lot of people who just don't care what X3J11 says; who believe that X3J11 is irreparably broken, and intend to ignore it into the indefinite future. It may well be that ``the ANSI C is already spoken.'' For my money, the ANSI C is already buggered. [snip] John ``Down With ANSI C'' Mackin, Basser Department of Computer Science, University of Sydney, Sydney, Australia john@basser.oz.AU (john%basser.oz@SEISMO.CSS.GOV) {seismo,hplabs,mcvax,ukc,nttlab}!munnari!basser.oz!john [snip] We seem in danger of heading the same way again. -- Joseph S. Myers jsm28@cam.ac.uk

Date: Tue, 06 Oct 1998 15:05:14 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> an API that does not provide this low-level control over timing hazards can[not] be useful for sub-nanoseconds measurements I agree, but it seems to me that an implementation should be able to supply such low-level control, perhaps as implementation-defined TIME_xxx flags passed to xtime_get. you will never want to use anything like xtime_get() itself as a source of sub-nanosecond timing information. I'm not sure that I agree with the ``never'', as clocks will keep getting better and better, and eventually (I hope within my lifetime :-) places outside the PTB will break the 1ns barrier. And I would like the spec to be modified so that xtime_get(TIME_MONOTONIC) can be compiled to a single instruction accessing the hardware clock register directly, which will make sub-ns access more practical. Also, we must distinguish between the overhead of calling xtime_get(), and the precision of the result. For some applications, it's enough that xtime_get() returns a precise value even if the amount of time that it takes to invoke xtime_get() is greater than the precision. All that's needed is that the clock had the reported value at some time during the invocation of xtime_get(). (Hmm, perhaps the spec should say something about this?) I am starting to get concerned that I might have wasted a lot of time while working on this, especially since I didn't get any feedback yet from the only ISO C committee member on this list. I also hope that your time wasn't wasted, but I don't think that the C9x is the only avenue for your work to be exploited. If we can get this into the GNU C library, for example, a lot of users will have it available in fairly short order. This will help convince the ISO C committee the next time around.

Paul Eggert wrote on 1998-10-07 09:01 UTC:
you will never want to use anything like xtime_get() itself as a source of sub-nanosecond timing information.
I'm not sure that I agree with the ``never'', as clocks will keep getting better and better, and eventually (I hope within my lifetime :-) places outside the PTB will break the 1ns barrier. And I would like the spec to be modified so that xtime_get(TIME_MONOTONIC) can be compiled to a single instruction accessing the hardware clock register directly, which will make sub-ns access more practical.
Don't forget that the old clock_t clock(void); is still available and can be used to provide direct access to the internal bus cycle counter. The xtime interface is intended for timestamps that have been converted to the SI second scale. Raw hardware clock counter values are outside its scope and are already supported by C89. So no need to change the proposal here. Just make in your implementation clock_t a 64-bit type, clock() a macro with inline assembler code to read the counter, and CLOCKS_PER_SEC a global variable that contains the current clock frequency. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Date: Wed, 07 Oct 1998 11:35:13 +0100 From: Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> Paul Eggert wrote on 1998-10-07 09:01 UTC:
I would like the spec to be modified so that xtime_get(TIME_MONOTONIC) can be compiled to a single instruction accessing the hardware clock register directly, which will make sub-ns access more practical.
Don't forget that the old clock_t clock(void); is still available It's available, but it wouldn't work well, for three reasons. First, `clock' reports the sum of the CPU times of the process and its children; this is not the same as TIME_MONOTONIC. Many applications need high-precision real-time clocks, not CPU-time clocks. Second, on my host (Solaris 2.6) `clock' is already defined to return a 32-bit microsecond counter, and it will be inconvenient to change this to a 64-bit or larger counter, due to backward compatibility concerns. Third, the struct xtime spec deprecates both clock_t and `clock'. One of the goals of struct xtime -- a goal that I agree with -- is to supersede `clock'.

In message <199810072258.PAA01063@shade.twinsun.com>, Paul Eggert <eggert@twinsun.com> writes
Don't forget that the old clock_t clock(void); is still available
It's available, but it wouldn't work well, for three reasons.
First, `clock' reports the sum of the CPU times of the process and its children; this is not the same as TIME_MONOTONIC.
clock should report the CPU time used by the process, *not* any other processes that happen to be related in some way. -- Clive D.W. Feather | Regulation Officer, LINX | Work: <clive@linx.org> Tel: +44 1733 705000 | (on secondment from | Home: <cdwf@i.am> Fax: +44 1733 353929 | Demon Internet) | <http://i.am/davros> Written on my laptop; please observe the Reply-To address

Paul Eggert wrote on 1998-10-07 09:01 UTC:
All that's needed is that the clock had the reported value at some time during the invocation of xtime_get(). (Hmm, perhaps the spec should say something about this?)
I certainly thought about providing some wording on the required or recommended relation between the timestamp and the entry/exit time of the function call, but I couldn't find a satisfactory formulation for which I didn't see problems. Some operating systems do variable time operations such as checking signals either before or after each system call, which make it difficult to get something widely implementable formulated her. If you need really precise guaranteed semantics, you probably will have to read the spec of your system and do it yourself in assembler. Portablility standards such as ISO C must have great limitations in what they are allowed to specify here.
I am starting to get concerned that I might have wasted a lot of time while working on this, especially since I didn't get any feedback yet from the only ISO C committee member on this list.
I also hope that your time wasn't wasted, but I don't think that the C9x is the only avenue for your work to be exploited. If we can get this into the GNU C library, for example, a lot of users will have it available in fairly short order. This will help convince the ISO C committee the next time around.
Once things have cooled down a bit, I'll probably write a paper about concepts for time APIs in general and might provide something like xtime (or an Ada version of it) as a discussion example. The heated discussion here indicates that this certainly is a not yet widely understood subject that deserves a bit of scientific discussion. And yes, the Linux kernel and GNU libc are certainly good demo implementation candidates. Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> writes:
I am starting to get concerned that I might have wasted a lot of time while working on this, especially since I didn't get any feedback yet from the only ISO C committee member on this list.
I am on the joint ISO/ANSI C++ committee. It's a safe bet that the C++ committee will be considering time and time zone representations for future technical reports, amendments, or revisions to the C++ standard. To be considered for addition to existing standards, a proposal like this must be implemented and demonstrated to be useful in real programs. Posting a library interface with arguments in its favor is not sufficient. Instead, rewrite demonstrably useful programs to use your interface, and show improvements in clarity, correctness, safety, and/or efficiency. Nathan Myers ncm@cantrip.org http://www.cantrip.org/

In message <199810052044.NAA14428@shade.twinsun.com>, Paul Eggert <eggert@twinsun.com> writes
Perhaps I'm paraphrasing you incorrectly; still, I am extremely uncomfortable with the idea that the C standard should impose an upper limit on the quality of the clock.
It shouldn't. However, that doesn't mean that it can't put a limit on Standard access to the clock. Thus a design that limited defined precision to 1ns, but didn't forbid an fs or as field would be reasonable. -- Clive D.W. Feather | Regulation Officer, LINX | Work: <clive@linx.org> Tel: +44 1733 705000 | (on secondment from | Home: <cdwf@i.am> Fax: +44 1733 353929 | Demon Internet) | <http://i.am/davros> Written on my laptop; please observe the Reply-To address

"Clive D.W. Feather" wrote on 1998-10-08 19:14 UTC:
In message <199810052044.NAA14428@shade.twinsun.com>, Paul Eggert <eggert@twinsun.com> writes
Perhaps I'm paraphrasing you incorrectly; still, I am extremely uncomfortable with the idea that the C standard should impose an upper limit on the quality of the clock.
It shouldn't. However, that doesn't mean that it can't put a limit on Standard access to the clock. Thus a design that limited defined precision to 1ns, but didn't forbid an fs or as field would be reasonable.
If there really is a hard-core fraction of attosecond fanatics in WG14, I'd be happy to replace xtime with struct xtime { xtime_sec_t sec; xtime_subsec_t subsec; } #define SUBSEC_PER_SEC ... /* subseconds per seconds */ I just tried to do the technically correct thing before the politically correct thing. Nobody is ever going to need subnanosecond resolution from xtime_get. Leaving the subsec encoding open will make portable programming more cumbersome and less readable. If you want to provide for direct hardware counter access (perfectly reasonable idea), then the "right thing"[TM] to do is to make an interface that looks precisely like clock() (just with a different name), but that gives access to an integer real-time counter instead of a process-time counter. This is what people interested in subnanosecond timing might want to use, but as I said before, these things are much more likely to be implemented in inline assembler or via direct I/O access anyway, because they are incredibly non-portable by their very nature. There is no chance to get even tens of nanosecond timing done in a portable way, so portable subsecond nanosecond timing really is science fiction. It is an "interesting" balance of design to require on one side the standard to provide provisions for unrealistic subnanosecond clock readings, and on the other size to provide no delay function at all (which will force authors of portable software to use inefficient wait loops that continuously read the clock until it is too late). But hey, if it makes the committee happy, ... Markus -- Markus G. Kuhn, Security Group, Computer Lab, Cambridge University, UK email: mkuhn at acm.org, home page: <http://www.cl.cam.ac.uk/~mgk25/>

In message <E0zP2OP-00036M-00@heaton.cl.cam.ac.uk>, Markus Kuhn <Markus.Kuhn@cl.cam.ac.uk> writes
Another problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it. This might best be another strfxtime directive.
Which would mean that we have to force the implementor to store the original string. Is this really necessary? The user has provided the string himself, so why whould he depend on getting it back later. We certainly could easily add another strfxtime conversion specifier, but I wonder whether this is necessary at all.
I used to think along those lines, but realized that it is wrong. The part of the program that sets the state might be totally unrelated to the part that needs to retrieve it (perhaps to override it temporarily). You can't rely on the two parts being able to be modified in sync. Nor do you need to store the original string. You just need to recreate one that will have the same effect. -- Clive D.W. Feather | Regulation Officer, LINX | Work: <clive@linx.org> Tel: +44 1733 705000 | (on secondment from | Home: <cdwf@i.am> Fax: +44 1733 353929 | Demon Internet) | <http://i.am/davros> Written on my laptop; please observe the Reply-To address

Markus Kuhn writes:
Nathan Myers wrote on 1998-09-16 19:10 UTC:
A problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it. This might best be another strfxtime directive.
Which would mean that we have to force the implementor to store the original string. Is this really necessary? The user has provided the string himself, so why whould he depend on getting it back later. We certainly could easily add another strfxtime conversion specifier, but I wonder whether this is necessary at all.
Yes, it's necessary. You say "the user", but there is not just one user. Each library used in a program may be written independently, and adding interfaces to each library to communicate the time zone name around a system separately of the time zone itself just adds clutter. Nathan Myers ncm@cantrip.org

Nathan Myers wrote:
Markus Kuhn writes:
Nathan Myers wrote on 1998-09-16 19:10 UTC:
A problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it.
Which would mean that we have to force the implementor to store the original string. Is this really necessary?
Yes, it's necessary.
What is the need? A string to be able to re-establish the same timezone later, perhaps in another program (operating on the same machine, of course)? Or really the same string used on input? The first API will return, as does setlocale() today, some string suitable for future use of tz_alloc. It requires a static back pointer in the internal datas. But it will return ":America/New_York" if you send "US/Eastern". Bad? Wrong? Right? The second will allocate storage to hold the string. And there will be no way to reclaim it when the timezone_t object will be freed. And also, this is some form of wasting in small implementations. If I see needs for the former, I have problems with the later. But I may be wrong. Also, the output of strfxtime is subject to localization. But the string passed to tz_alloc is not, IMHO, a data that *should* be localized. How should that be handled? Antoine

Antoine Leca writes:
Nathan Myers wrote:
Markus Kuhn writes:
Nathan Myers wrote on 1998-09-16 19:10 UTC:
A problem I foresee is that there is no way, given a timezone_t object, to retrieve the string used to construct it.
Which would mean that we have to force the implementor to store the original string. Is this really necessary?
Yes, it's necessary.
What is the need? A string to be able to re-establish the same timezone later, perhaps in another program (operating on the same machine, of course)? Or really the same string used on input?
Any string that will re-create the same time zone object is sufficient. The string used to construct it is the simplest to (re-)produce.
The second will allocate storage to hold the string. And there will be no way to reclaim it when the timezone_t object will be freed.
This does not follow. When the timezone_t object is freed, the string storage can be freed as well. Note that strfxtime (or, better, xtime_format) does not return pointers to internal storage.
Also, the output of strfxtime is subject to localization. But the string passed to tz_alloc is not, IMHO, a data that *should* be localized. How should that be handled?
So, then don't localize it. Or define "localizing it" to mean nothing is changed in it. I don't see the problem. Nathan Myers ncm@cantrip.org

In message <13823.62501.12960.906883@localhost>, Nathan Myers <ncm@cantrip.org> writes
I don't like to see the %H, %M, and %S formats restricted in their format to only '.' and ',' decimal separators. Separate directives (perhaps %.nH et al) that format only the fractional part would allow users to supply any decimal separator they chose, as in "%H:%M!%.3M" for "03:20!666". (I have seen satellite navigation systems with stranger choices of syntax.)
I think you're barking up a tree that WG14 won't like here. The strftime() family of functions are designed to extract the basic time information in text form. I previously attempted to get minimum widths and zero fill added, but unsuccessfully. The consensus view of WG14 was that the programmer should do text manipulation on the results of the conversion. I don't think this standard is the place for unusual features like decimal times, let alone specified precisions. -- Clive D.W. Feather | Regulation Officer, LINX | Work: <clive@linx.org> Tel: +44 1733 705000 | (on secondment from | Home: <cdwf@i.am> Fax: +44 1733 353929 | Demon Internet) | <http://i.am/davros> Written on my laptop; please observe the Reply-To address
participants (9)
-
Antoine Leca
-
Clive D.W. Feather
-
D. J. Bernstein
-
John Cowan
-
Joseph S. Myers
-
Ken Pizzini
-
Markus Kuhn
-
Nathan Myers
-
Paul Eggert