On 3 September 2013 09:15, Brian Inglis <Brian.Inglis@systematicsw.ab.ca> wrote:

On 2013-09-03 01:34, Paul Eggert wrote:

...

Link to America/Port_of_Spain instead of having a separate zone that differs only in LMT. Each LMT entry is just a placeholder for unavailable info, and by itself does not justify a separate zone. Using just one zone simplifies maintenance, makes the runtime a bit smaller, and can help simplify user selection of zones. * southamerica (America/Aruba): Likewise, for America/Curacao. * backward (America/Virgin): * northamerica (America/St_Barthelemy, America/Marigot):

Folks seem to have forgotten that initial LMT entries contain the Standard Time start date, as well as the prior GMTOFF based on longitude of that location.

While I agree that merging zones with different GMTOFF makes no significant difference, merging zones with different Standard Time start date loses useful historical data, unless the tzdb is seen mainly as a source of DST changes since 1970.

In the changes highlighted below, Standard Time start dates vary from 1890-1921, with some different dates in 1911-1912.

Zones should be merged only if all linked locations have the same Standard Time start date, if we wish to maintain the history of when Standard Time was established in each time zone.

Yes, unfortunately I agree with Brian. I note that the patch also changed the observable time-zone abbreviations (KMT/CMT etc). Again, while this might be invented data, through existence over a long period it has become undeletable. As such, most of this patch would need reverting :-( Stephen

On 3 September 2013 15:47, Paul Eggert <eggert@cs.ucla.edu> wrote:

Brian Inglis wrote:

...

merging zones with different Standard Time start date loses useful historical data

That's not a problem for the proposed change, because in this particular case, no useful historical data are lost. There is no real evidence that the start dates actually differed. All we have is some guesswork from Shanks. The Shanks data often contain guesswork, and the abovementioned transition dates from LMT are almost certainly part of that guesswork.

The associated time zone abbreviations are also guesswork (in this case, my guesswork and not Shanks'), so they do not contain any useful historical data either.

Your long experience with the tzdb leads you to assert that these are guesswork, therefore OK to delete. Unfortunately, the consumers of the tzdb see the data as fact, and since the data has been there for many years that simply reaffirms the "fact" status. I'd also suggest that Shanks information has at least some degree of thought behind it, certainly better than picking up the thoughts of some other location that has always been under another jurisdiction. I really wasn't joking when I suggested that pretty much no refactoring is now possible. Its certainly the case that pretty much every refactoring you can think of has some data loss. thanks Stephen

Paul Eggert wrote:

Stephen Colebourne wrote:

...

...

Shanks information has at least some degree of thought behind it, As a longtime reader of the Shanks data, I have a good feeling for when it's reliable and when it isn't. This is definitely a place where isn't.

Followers of Time Team will recognise this :) Nice drawings of some ancient site which when reopened bears no relation to real remains ...

...

...

the consumers of the tzdb see the data as fact That's a problem, since much of the data are wrong. And it's all the more reason to omit dubious data when we can, which is the case here.

Am I detecting something of a pattern here ... if material is inaccurate or unsubstantiated, then fine it needs to be removed. The git version control is not ideal for tracking this type of update but it does at least retain the history of corrections and the commit message should explain the detail. But in keeping what remains, can we at least agree that the quality of the material is heading in the right direction? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

On 3 September 2013 16:52, Paul Eggert <eggert@cs.ucla.edu> wrote:

Stephen Colebourne wrote:

...

the consumers of the tzdb see the data as fact

That's a problem, since much of the data are wrong. And it's all the more reason to omit dubious data when we can, which is the case here.

If you were just removing incorrect data, then that would be an enhancement. But you are in fact removing one set of dubious data and replacing it with another set of dubious data from an entirely different location, via the link. Thats a problem (not an earth shattering one, but its still negative, not positive). Whereas if you did nothing, made no change, then we would't be having this discussion. It is my judgement that this change, like many recently, are not what I would describe as enhancements - they are just refactoring. Its your desire to refactor that is at the heart of pretty much every recent thread about reverts. If you had made no changes since May, apart from to the C code and to current local time, then I, and many others, would be entirely happy. Stephen

On 3 September 2013 22:27, Russ Allbery <rra@stanford.edu> wrote:

Stephen Colebourne <scolebourne@joda.org> writes:

...

If you had made no changes since May, apart from to the C code and to current local time, then I, and many others, would be entirely happy.

I've been sitting on this comment for quite some time, but, seriously, haven't you noticed that it's trivial for you or anyone else to produce a tz distribution that satisfies these requirements? If that's what you want, well, you have all of the tools required to do this right now, and could have been doing this since May with far less investment of effort than you're currently putting into long messages to this mailing list.

I guess I don't understand why you seem so deeply invested in convincing Paul that you're right and he's wrong when it's fairly trivially possible for you to generate exactly the data that you want according to the criteria that you want followed without insisting that other people do the work for you.

If you want to freeze all historical data in stone at the state they were in as of May of this year and only adopt forward-looking changes, then by all means do so! Set up a web site, set up a Git repository, cherry-pick the changes you want, and enjoy the perfect control you then have over your data source.

I have been following this mailing list for many, many years, and when I first started following it I read all messages to this list going back to the foundation of the list. With that information in mind, I'm quite comfortable in saying that the maintenance policy that you're asking for is not the maintenance policy that ado used, and is not the maintenance policy that has ever been used for this project. I don't understand why you're so insistent on pushing a different maintenance policy on the project against multiple objections instead of just filtering the project changes down to the ones you approve of and publishing your own work.

There are a number of points here. Firstly, creating a fork of the data is entirely unsatisfactory. Alongside stability, I have indicated that ubiquity is a key value of the data. The same data is used everywhere from Unix to Java to mobile phones. Creating a fork greatly devalues the data (the value of the two parts is less than the value of the whole). In addition, the tzdb charter requires changes to have consensus, so I have every right to complain here. Secondly, I'm not speaking on behalf of myself, but on behalf of Java development generally. I'd also suggest that I'm expressing the opinions of enterprise software development more generally (see the comments to the list from Bloomberg and Oracle RDMS). Thirdly, I note that the leading supporters of Paul's approach are from an academic background (Paul, Guy, yourself). With respect, I wonder if that academic background insulates from the needs of enterprise software, primarily stability. Fourthly, it seems to me that the recent batch of changes are far in excess of what has happened over previous years. For example, https://github.com/eggert/tz/commits/master/backward shows that the backward file was modified a number of times in the past few years, but almost always for changes to the spelling or naming of zone IDs (something which I've not opposed, even though I know CLDR finds that problematic). Finally, I'm NOT asking for all historical data to be frozen. I'm asking for no historical data to be changed UNLESS the replacement is a clear enhancement. This is a common sense and perfectly reasonable request of the database. Its also how all well-resected APIs and data sets (like CLDR) operate. Such an approach does explicitly rule out zone ID merging that loses the start date of offsets or abbreviations, even if those are guesswork/invented (because the replacement is not an enhancement, its worse). To be clear, I hate the fact that I'm having to write these emails. I think this list should be incredibly quiet, only receiving an email when a current time changes, or when someone does some historical research into a zone. Its not my fault that the database maintainer is insisting on making uneccessary and negative changes, and then refuses to revert them when the lack of consensus is clear. In summary, given the importance of the data set, and how it is currently being abused, I have no choice but to pusue my objections. Stephen

On 4 September 2013 15:32, Paul Eggert <eggert@cs.ucla.edu> wrote:

...

ubiquity is a key value of the data. The same data is used everywhere from Unix to Java to mobile phones.

No, it's pretty routinely filtered before it hits many platforms. One example: QNX has unsigned time_t, which by design filters out all data before 1970.

Furthermore, there is an inevitable delay in propagating changes to the field. Even if we're talking a single host with 64-bit signed time_t (so that it matches Java's 'long'), I've seen situations where Java's copy of the data disagree with the POSIX copy. And certainly a distributed application cannot assume ubiquity, as the client and server may be updated at different times. So, for various reasons unrelated to the proposed changes, it's already the case that applications cannot assume that the data are ubiquitous and that the same data are used everywhere.

Not the ubiquity I meant. I meant that if there exists a fork of tzdb, then it will over time diverge. If Scotland has its own time different from England, then one tzdb might name it Edinburgh and the forked tzdb uses Glasgow. That divergence, or non-ubiquity, would be very unhelpful to everyone that needs time-zone data.

...

I'm not speaking on behalf of myself, but on behalf of Java development generally. These comments would have more weight if they pointed to user problems that occurred when we made similar changes in the past. Based on my experience I'm skeptical that there were significant user problems. I've asked the list for reports of problems but nobody else has reported problems either. This suggests that the concerns are misplaced.

The data is used far more widely that just zic. Some of those uses work directly from the source tzdb data. Some of those uses expose source tzdb data that is not exposed via the zic binary. Thus, it certainly is the case that there are people affected by each change. Chances are they won't know it until a month or two down the line. For example, the removal of "Castries Mean Time" and "Kingstown Mean Time" will be visible in Joda-Time, and the change to the end of LMT will be visible in Joda-Time and JSR-310. Most people will adapt, not complain. But it is clearly a fact to say that data has been deleted and that deletion is observable to consumers of the data. The only argument is whether that data was sufficently in error to warrant deletion.

On this list I have also noted that the changes promise to make life easier for users in some cases, by omitting irrelevant choices. This is a real advantage that should trump stability concerns.

There are other ways (winnowing) to reduce the selection problem, because that problem is based off zic. For those of us parsing the source tzdb files directly, any data loss is data loss.

...

zone ID merging that loses the start date of offsets or abbreviations, even if those are guesswork/invented (because the replacement is not an enhancement, its worse).

I've had quite a bit of experience in dealing with the Shanks data. From my experience the proposed change is a fairer representation of what we know than the previous version was. You're right that we don't know that the new version is correct and the old is wrong (both are guesses), but it's not right to say that the new version is worse.

Changes like this are a longstanding part of maintenance, and I'm becoming inclined to think that we shouldn't discontinue this practice purely from a desire to not change things.

Again, I don't want to stop enhancements, that would be counter-productive to all. However, by making the change you are asserting that it is more true that all 10 locations in the Carribean have the same local time since the year dot, than that they had different time (that we just don't know anything about). ie. the fact that you have made a change at all means that the data should now be more reliable. Yet you accept that both the before and after are guesses. So, if you can reply to this email and say "with additional research I can say with reasonable certainty that all 10 locations have always had the same local time" then the change is entirely justified. If you can't, then the change should be reverted as not based on enough evidence. (To put it another way, the barrier required for changing the ID like this is higher than the barrier was when the ID was created in the first place. For example, America/Curacao and America/Aruba have exactly the same time since the year dot apart from the LMT value (they do have the same LMT end date). As such a Link is appropriate. Whereas, America/Tortola and America/Port_of_Spain have different LMT end dates. This means that Joda-time users will see a change in local time when querying between 1911-07-01 and 1912-03-02. More broadly, I'd suggest it would have been wiser to only suggest this patch once the current mess has settled down. Stephen

On Wed 2013-09-04T11:01:24 -0700, Paul Eggert hath writ:

But ordinary users won't care that time stamps in Aruba on February 12, 1912 from 04:35:47 to 04:40:24 UTC will have a UTC offset that differs by a few minutes. They just won't.

I request caution in making it clear to ordinary users that the name UTC cannot be proleptically extended to dates prior to 1960. No such concept existed in contemporary records. -- Steve Allen <sla@ucolick.org> WGS-84 (GPS) UCO/Lick Observatory--ISB Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m

On Wed 2013-09-04T20:43:18 +0100, Lester Caine hath writ:

...

I request caution in making it clear to ordinary users that the name UTC cannot be proleptically extended to dates prior to 1960. No such concept existed in contemporary records.

Being pedantic :) 1963 ... But none of the sources actually quote the date for Recommendation 374 :(

I have examined the pages and kept images of that document. http://www.ucolick.org/~sla/leapsecs/timescales.html The 10th Plenary Assembly of the CCIR was held in Geneva from 1963-01-15 to 1963-02-16. During that meeting Recommendation 374 was approved to supersede Recommendation 319. Recommendation 374 does not contain any form of the words "coordination", "coordinated", nor the letters "UTC". The text states that the broadcasts of time signals should "be offset to keep the time pulses in close agreement with UT2" and "maintained within approximately 100 ms of universal time UT2" with steps of exactly 50 ms. -- Steve Allen <sla@ucolick.org> WGS-84 (GPS) UCO/Lick Observatory--ISB Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m

On Wed 2013-09-04T14:40:57 -0700, Paul Eggert hath writ:

Yeowch! You're right; sorry about that. I should have written "GMT". I was misled by zdump's output.

Except that GMT prior to 1925 means one thing for civil timestamps and a different thing for nautical timestamps.

Should we change the output of "zdump" etc to fix this error? Currently zdump says "UTC" for old time stamps, which isn't correct. Should it say "UT" instead? Or is even "UT" a bad idea for a time stamp in (say) 1627?

Unlike UTC, the concept of UT can be validly extended into the indefinite past.

I also should have mentioned that even with GMT, my comment was incorrect in some sense. Common practice back then for Dutch possessions was to use non-integer offsets from GMT, and the tz format cannot represent these. I don't have good data for Aruba, but Capt. Thomas Henry Tizard of the Royal Navy reported that Curacao's port kept time at -04:35:46.9; see Milne 1899.

One might argue over the validity of the conventional formula for UT being used in the far past, and for the purposes of geodesy it is relevant to be precise, but to the relevant precision for civil timestamps any reasonable conventional expression for UT is as valid as the timestamps themselves. -- Steve Allen <sla@ucolick.org> WGS-84 (GPS) UCO/Lick Observatory--ISB Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m

On Wed 2013-09-04T19:09:52 -0700, Paul Eggert hath writ:

Steve Allen wrote:

...

Unlike UTC, the concept of UT can be validly extended into the indefinite past.

Thanks for bringing this up. Here's a proposed patch that tries to fix the occurrences of this problem that I found in the tz code and data. I've pushed this to the experimental repository.

These patches make my POSIX conformance side very uncomfortable. The comments about the data are generally more accurate by using UT. The comments in and about the POSIX API get very confusing for not using UTC (as is done in the POSIX docs themselves), and I think they might best be left referring to UTC. Unfortunately there is no single good answer for the name of the Greenwich-like time scale that is presumed by computer APIs. In the absence of switching the zdump output string between UT and UTC as of 1960 some sort of apologetic explanation may be required. -- Steve Allen <sla@ucolick.org> WGS-84 (GPS) UCO/Lick Observatory--ISB Natural Sciences II, Room 165 Lat +36.99855 1156 High Street Voice: +1 831 459 3046 Lng -122.06015 Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m

On Sep 4, 2013, at 2:32 AM, Stephen Colebourne <scolebourne@joda.org> wrote:

Thirdly, I note that the leading supporters of Paul's approach are from an academic background (Paul, Guy, yourself).

Actually, no, I'm not "from an academic background", except to the extent that I have an academic degree; everywhere I've worked since 1979 was a business of some sort, not in academia. ("alum" is short for "alumni"/"alumnae", if you're guessing from my e-mail address that I work in academia; it's a mail-forwarding service.)

With respect, I wonder if that academic background insulates from the needs of enterprise software, primarily stability.

I've worked as a software developer at various companies, such as Sun Microsystems, Network Appliance, and Apple; only one of them was a major producer of "enterprise" equipment when I worked there (Sun was a technical workstation company, not "the dot in dot com", then, and Apple hasn't been particularly enterprise-oriented in ages). As for whether I'm a "supporter of Paul's approach", I think that's an overstatement. I'm somebody who's certainly sympathetic to concerns about having to offer end-users choices between tzids that won't make any real difference to them in practice - i.e., I'm not 100% an *opponent* of Paul's approrach - but I think the *ideal* way to handle that is not to offer end-users choices between tzids, which is why I've cited OS X's "Time Zone" subpane of "Date & Time" in System Preferences as the right way to handle that, rather than the cheesy "I know! I'll just turn all the underscores in tzids into spaces and make an option menu out of them!" stuff done by some other OSes. There are now data files freely available that can support that. For developers that are hard-coding tzids into code or files, they'd have to know what they're doing and pick the appropriate tzids. For those letting the users specify a tzid, something OS X-style would be best, so they don't need to know about tzids.

Finally, I'm NOT asking for all historical data to be frozen. I'm asking for no historical data to be changed UNLESS the replacement is a clear enhancement. This is a common sense and perfectly reasonable request of the database. Its also how all well-resected APIs and data sets (like CLDR) operate. Such an approach does explicitly rule out zone ID merging that loses the start date of offsets or abbreviations, even if those are guesswork/invented (because the replacement is not an enhancement, its worse).

I personally think it reasonable that if two entries differ in the start date for standardized times, *and* the start dates have a reliable source for them, they not be coalesced. If the start dates are just guesses, that sounds like a change from one set of data that won't necessarily give you the same answer to another set of data that won't necessarily give you the same answer, so I don't think it's clear that merging them is a bad idea - stability might be an argument against merging them, but the only way in which the old data could be considered "better" is that it doesn't represent a change, not that it necessarily better reflects reality. I.e., in the case you mention, the replacement is "worse" only in that it's different, not in that it's further from reality. And I think anybody using the tzdb to handle pre-1970 dates, or claiming that their APIs can be used to handle pre-1970 dates, should either state it as "you can use it, but we make no claim that the results will be correct, so don't rely on it for anything where a difference of an hour or two will actually matter" or should back up their confidence with historical research (and send the results of that research to Paul so the comments can be updated).

Stephen Colebourne wrote:

...

I'll just turn all the underscores in tzids into spaces and make an option menu out of them!" stuff done by some other OSes. There are now data files freely available that can support that. Moving on from my dumb academic statement, I agree entirely that a good user interface should not expose raw or mangled raw zone IDs. CLDR offers good textual forms for example.

Or after asking their home country, only add a timezone selection if that is necessary? I think that the naming of zones is something of a smoke screen, although we do need a good cross reference as to what location uses what timezone - and when, and on edge cases it may well be difficult for a local to pick which of two time zones is right for them? It is the quality of the data that is the first problem, and making sure that what is presented is as complete as possible rather than saying 'It's too difficult so we will not try'. The best people to research local data are the locals themselves and while some parts of the world documentary evidence may be none existent, even that is a fact that should be advised! Basically despite previous best practice, there is now a growing need to complete the picture but it does require the assistance of the world in general to do that. -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

Stephen Colebourne <scolebourne@joda.org> writes:

Secondly, I'm not speaking on behalf of myself, but on behalf of Java development generally.

That's quite a mantle for you to be assuming for yourself. I have to say that I would be very reluctant to make a similar claim. I'm not sure I'd want to be responsible for expressing the opinions of the entire Java development community as a self-appointed spokesperson.

Thirdly, I note that the leading supporters of Paul's approach are from an academic background (Paul, Guy, yourself). With respect, I wonder if that academic background insulates from the needs of enterprise software, primarily stability.

I'm not that horribly interested in a war of credentials, but perhaps some additional professional background would be helpful for context. I am, professionally, an IT architect and software developer for the central IT department at Stanford University. I'm not sure whether you count that as an academic background or not, but my day job is running a large hetergeneous server infrastructure. My professional specialty is authentication systems, which tend to care very deeply about accurate time, but my involvement in this mailing list is out of personal interest and curiosity. I am also a member of the Debian Technical Committee and one of the editors of the Debian Policy documentation, so I'm reasonably familiar with distribution packaging issues, although I am not personally involved in the Debian packaging of the tz database or in time zone selection during the Debian installation process. That said, I want to make it clear that I don't speak for Debian, let alone for Linux packagers in general. I'm participating, as I hope all of us are, as an individual with some interest in helping the project make the best decisions that we can make, and to provide support to Paul as the primary maintainer.

Fourthly, it seems to me that the recent batch of changes are far in excess of what has happened over previous years. For example, https://github.com/eggert/tz/commits/master/backward shows that the backward file was modified a number of times in the past few years, but almost always for changes to the spelling or naming of zone IDs (something which I've not opposed, even though I know CLDR finds that problematic).

I think it's important to distinguish between two different things that are happening. There have been quite a few changes made recently on a trial basis in an attempt to address some of the geopolitical concerns. I have no specific comments on that other than to say that I wholeheartedly approve of and support the *process* that Paul has been using in trying to reach consensus on how to address those problems, including floating trials and then backing them out when people disagree with them. I'm personally frustrated by people treating every proposed change as if the world might end; by all means, argue your side of this debate if you have strong opinions, but some of the comments have bordered on accusing Paul of acting in bad faith, and that's not sitting well with me. I have my own opinions about the origin of the recent flood of geopolitical concerns (and they're much harsher than Paul's, which is one of the reasons why I've been sitting on my hands and letting Paul handle it, much better than I would have). But I think it's important to remember that Paul is making a good-faith effort to address issues that have been raised, and to act and debate accordingly. However, apart from that set of changes, from where I sit, you and a small number of other people have gone beyond that argument and have now started objecting to nearly every change Paul makes for any reason, including changes that would have been entirely uncontroversial in previous years. And that's what I'm taking exception to.

Finally, I'm NOT asking for all historical data to be frozen. I'm asking for no historical data to be changed UNLESS the replacement is a clear enhancement.

The bar that you're setting for "clear enhancement" is not consistent with how this project has ever been run in the past.

In summary, given the importance of the data set, and how it is currently being abused, I have no choice but to pusue my objections.

You are bringing far too much drama to this situation, in my opinion. Paul has demonstrated repeatedly that he's not only open to reasonable discussion, he's open to reverting changes even when people aren't able to express coherent objections but are just upset. None of this is the end of the world. There are some very hard problems around the intersection of geopolitics and time zone selection, but there are also multiple layers of correction and UI between the core database and those issues. And the changes that people are worried about have not been in any official release. I understand why you want there to be a single tz database run according to your criteria, but that isn't an option on the table. There can be multiple tz databases, of which one is run according to your criteria, or there can be one database where you provide only one point of input among many other people. The contributions you have brought are very valuable. It is very useful to have someone deeply involved in the project who understands and cares about Java's use of the database, particularly since some of those uses are quite different than the typical POSIX use of the database (which was its original raison d'etre and still tends to shine through, although the project has grown beyond that). I would certainly prefer for you to continue to collaborate here. But from where I sit, the recent discussions have felt like more of a hostile takeover than a collaboration, particularly when you casually dismiss all the work Paul has done over the past several months. That is what prompted me to speak up. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>

Stephen Colebourne <scolebourne@joda.org> writes:

Part of the problem is that commits to the repo form the basis of the permanent record of the group. IMO, if the commits are experimental, then they should occur first on a branch, and only be included onto the master branch once they are accepted. With the current way of working, someone in 5 years time is going to have to wade through a lot of rubbish over the past month to see what really happened.

I think you're reading too much into this. The use of any modern VCS for this project at all is quite new; prior to the change of maintainer, it was maintained in SCCS and the SCCS files were not even generally public. There are many possible workflows with Git, and they are, to a large extent, a matter of personal opinion and personal comfort. I probably would have used a branch, but I might not have, and in any event I don't think it's particularly important. I would certainly not object if Paul decided that maintaining a high quality commit stream with rich metadata was a maintenance standard that he wanted to adopt for the project, but to me this is a lot more about maintainer workflow than a vital output product of the project, and if it's not work he wants to do, oh well. We will all certainly survive, just as we survived just fine through years of ado using private SCCS files. I think your second point here is more on point:

Secondly, I would be less peeved if reversions were actually reversions. Each reversion that has occurred has been partial rather than complete, or has included some other unrelated change. A much better practice would be to revert in full, and then reapply a smaller change with the hope that may be more acceptable.

I do think that the way the changes were made and reverted has made it more difficult to understand what portion of the change was reverted and what portion of the change was not reverted. It's fairly easy for anyone to determine the remaining changes by doing a git diff across the relevant objects, but I do concur that using the git revert command to back out the whole change and then applying the change that one wants to retain as a separate change is cleaner and easier for everyone else to understand. This has the advantage of providing a place to put rationale for the retained change separate from rationale for the original change. It is, however, inconsistent with a GNU ChangeLog style of documentation of project history, at least as I understand the ChangeLog standard. It's much more of a "native Git" way of handling things.

When the database moved to IANA, I argued that it should have gone to CLDR. They have a full technical architecture committee and a team of people used to dealing with complex political and i18n issues backed by enterprises such as IBM and Oracle. If it had gone there, the stability that I am calling for would have happened.

I'm very glad this didn't happen. I would have had much less respect for and much less faith in that sort of process. The tz database has always been a labor of love by a small number of technical professionals acting on their own time and with their best personal judgement. It is, in that way, either a throwback to a different era of computing or a participant in the free software world, depending on your preferred way of looking at things. Personally, I believe that's one of the major reasons why it has been so successful over such a long period of time.

For the record, I am not objecting to every change, I am objecting to every change that actively deletes data in a way that can be observed by a consumer, unless such a change is a clear enhancement.

It is certainly within your right to make that objection. I personally disagree with you, and am taking this opportunity to say so. I do not believe that is the correct standard to use when deciding on changes. I prefer the standard that has been used by Paul, ado, kre, and others in the past, which takes into account stability, accuracy of data, proliferation of zones, and ease of future maintenance to arrive at a criteria with more nuance than that. The policy that I'm speaking up to advocate for is that the people doing the work should have the most weight in deciding on the strategy that's the most effective. They're the ones bearing the burden of the work, and they're also the ones with the most experience with the data and the most understanding of which data points are significant and which are not. This, like most software projects, is as much art as science; subjective judgements matter, and are the path to increased quality and consistency when the same judgement is consistently applied. Putting aside the issues around tzselect and geopolitical sorting for the moment, most of the other changes that have happened recently would probably have never been noticed by anyone who wasn't reading the commit stream for the project. There seems to be quite a tempest in a teapot here, at least from my perspective.

I don't consider a request for stability to be a hostile takeover, but I do consider some of the proposed commits to be far in excess of what is acceptable to me.

I hope you can understand why I find phrases like "far in excess of what is acceptable to me" to be hostile and confrontational, and not helpful in reaching a collaborative consensus. That's a bargaining tactic in a hostile negotiation, which I would really hope we could avoid. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>

On Sep 4, 2013, at 2:23 PM, Garrett Wollman <wollman@csail.mit.edu> wrote:

I do think, on a more general level, that it is a bit unfortunate that the format of the tzdata files is now effectively frozen because so many external users have chosen to parse the source rather than the compiled form. This will make it much more difficult to make changes in the future, should they be needed.

If we want to do that (e.g., if we were to introduce the CLDR concept of meta-zones into the tzdb, so that the CLDR doesn't have to duplicate transition time/date information that's in the tzdb - see the usesMetazone items in metaZones.xml), we might end up creating a new source file format and have a tool that reads the new format and emits the old format.

On Sep 4, 2013, at 2:36 PM, Stephen Colebourne <scolebourne@joda.org> wrote:

Just to note that I parse the source tzdb because the binary data does not contain all the data AFAIK.

Which data is that? Perhaps we should see if we can extend the binary data format (in a backwards-compatible fashion, so that old code can read the new files) to include that data.

Stephen Colebourne wrote:

Notably, I don't believe that the binary data provides the rules themselves which allow future DST to be calculated.

The last item in a version-2 tzfile is a POSIX (System V style) TZ string to be used for the indefinite future of the zone. My tzfile parser (the Perl module) makes use of this. There are a small number of zones for which such a TZ string can't be formulated, and in those cases zic tries to provide 400 years of explicit future transitions, so if the zone's rules are based on the Gregorian calendar you can repeat that 400 years indefinitely. There are an even smaller number of zones for which neither of these strategies work, but in those cases the zic source can't represent the rule either. (Incidentally, I've posted a patch more than one that would make the 400 year hack more robust, and it seems to have got lost each time. It's sitting on a branch in my public git repo.) The extra information not in the tzfiles that you can get from the zic source is limited to things like whether a transition was the result of a long-term rule as opposed to a one-off decision, and what the rule was. I don't think this is really meaningful information for the purposes for which the zone data is intended. It also doesn't look as if previous practice has tried to maintain this information in a meaningful way, other than perhaps in the comments. For example, the UK during the 1980s and early 1990s had DST dates officially set separately by Statutory Instrument for each year, but the individual decisions tended to follow a consistent pattern that the "europe" file describes by three multi-year "Rule" entries. -zefram

Zefram wrote:

I've posted a patch more than one that would make the 400 year hack more robust, and it seems to have got lost each time. It's sitting on a branch in my public git repo.

Sorry about that, it was still in my inbox. I revisited it today. It inserts an artificial mark into the zic binary file so that, for zones in which POSIX TZ strings cannot support time stamps into the indefinite future, a consumer of the binary file can more easily determine where its data are valid. This jogged my brain to look two other items in my inbox that are relevant and less problematic. I fixed these and pushed the fixes into the experimental github. First, "Record that San Luis is at UTC-3, not UTC-4 with perpetual DST." <http://mm.icann.org/pipermail/tz/2013-September/019996.html> adjusts Argentina/San_Luis to not have a weird setting of perpetual DST indefinitely into the future, a setting that POSIX TZ strings cannot represent. Second, "Support time stamps past 2038 in zones like America/Santiago" <http://mm.icann.org/pipermail/tz/2013-September/020013.html> fixes zic to generate TZ strings that work into the indefinite future for almost all zones where this was a problem, thus removing the need for the artificial mark in these zones. There's one remaining zone which has the problem, namely Asia/Tehran, but I don't think your 400-year change fixes it. So, I'm hoping that your fix is no longer needed, in that the other two patches have fixed things in a different way. Anyway, thanks for bringing this up, as it prompted fixes for the problems in the database or the way it's processed on POSIX platforms.

Zefram wrote:

If zic is to retain the logic for the 400 year hack, it ought to be the robustified form of the hack.

Fair enough. But then I have a question about the change. It increases the window from 400 to 402 years. Is that part of the change needed? As I understand it, it's to avoid coalescing (say) a 399-year run of a rule to an adjacent one-off that *happens* to look like the extension of the rule. But is there really any harm to that? Such coalescing is what we already do, when preparing the input data.

On Fri, Sep 6, 2013, at 11:38, Zefram wrote:

You misunderstand it. The situation of concern is where the 400 year period governed by a rule is immediately preceded by something that is *different from* what the rule would have. If the preceding DST behaviour has transitions later in the year than the transitions produced by the rule, the last preceding transitions could occur less than 400 years before the last rule-generated transition that zic puts in the output.

Why not put a non-transition (i.e. a transition to the same offset already in effect) at the beginning and end of the 400 year period?

On Wed, Sep 4, 2013, at 17:36, Stephen Colebourne wrote:

Just to note that I parse the source tzdb because the binary data does not contain all the data AFAIK. A new source format that generates a file in the old format would work so long as the distribution contained the generated old format files.

The only data that the binary data doesn't contain is the question whether some transition was based on a "last sunday" rule or a fixed date, which is an entirely academic question for past data, and which it actually _does_ usually contain for the present/open-ended-future data (where it doesn't, it attempts to include a full 400 years on the assumption that the transitions will repeat exactly after 400 years) It's not clear why you need that information. Is it possible that you made this design transition before the 64-bit format was added, and needed to support dates after 2038?

I do think, on a more general level, that it is a bit unfortunate that the format of the tzdata files is now effectively frozen because so many external users have chosen to parse the source rather than the compiled form. This will make it much more difficult to make changes in the future, should they be needed.

If my understanding is correct, the output from zic does not preserve daylight saving amount. TimeZone class in our project (ICU) exposes an API for accessing base offset and daylight saving amount separately for historical reason (same thing found in JDK Calendar class). This is one of the reasons why we decided to customize zic and use the text source format as input. I personally think consumers of our code don't care about daylight saving amount (mostly +1 hour, but few exceptions) although. -Yoshito

( Written earlier but not sent :( ) Stephen Colebourne wrote:

I don't consider a request for stability to be a hostile takeover, but I do consider some of the proposed commits to be far in excess of what is acceptable to me. For the record, TZ Coordinator is not a job I would enjoy.

I'd second that, but personally I feel that there is still a difference of opinion as to the target we are trying to reach? 'Simplifying' the data may be removing material of a questionably origin, but it would be nice to see a little more substance to justify that? My own trawling through possibly supporting some of Paul's hatchet, but I am still uncomfortable that historic records are being lost and that there is still an underlying design to ignore pre 1970 data when to do so creates an area of the calendar where we can't rely on consistency between operating systems? If culling pre 1970 data is allowed then we end up with a second class view of the data in some environments? I now have a local copy of the git repo and can browse at leisure and One thing that DVCS is very good at is providing a platform to play and experiment. I do agree with Stephen that the correct action once the initial re-factoring was objected to should have been to roll back to a mutually acceptable version. At that point changes could have been cherry picked and those that were agreed reapplied. Where data is then removed it could be individually committed and tagged. And to further tidy up the repo ... moving the data to it's own repo will keep data commits away from code ones? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

On 4 September 2013 22:42, Paul Eggert <eggert@cs.ucla.edu> wrote:

On 09/04/13 13:08, Stephen Colebourne wrote:

...

commits to the repo form the basis of the permanent record of the group.

No, the github repository eggert/tz is my personal experimental version. The permanent record of this group (including all the diffs sent via email) is published by the IANA at <http://www.iana.org/time-zones>.

I don't see a git repo there (or any other kind of repo). If there isn't such a thing then there should be one. Patches on the mailing list are insufficient to keep track of the changes as recently shown.

The recent flurry of detailed comments about relatively minor points in the experimental version has tempted me to go back to how Arthur David Olson did it, which was to not publish experimental versions.

Its only by having such a git repo that I have been able to meaningfully criticise the changes and figure out what is going on. The problems stem from changes that were not accepted.

...

A much better practice would be to revert in full I suspect that we're looking at a clash of maintenance styles here. I prefer not to create clones of earlier versions (if you want the earlier version, it's easy enough to find it), but other people do. No matter which style is chosen, people who prefer the other style will find it irritating.

As an open and public project, it should be possible and desirable to have external people review things. The incomplete reversions made life extremely difficult. What I was asking for was that you do a full git revert for the original commit, and then add a second commit with the more accepted changes. That way if the second "fixed" commit turns out to need to be reversed then that can be easily seen as well. Git is all about lots of small commits. Stephen

The Shanks data often contain guesswork, and the abovementioned transition dates from LMT are almost certainly part of that guesswork. ------- This is true. Gary Christian (CEO Astrolabe) has admitted that time zone info represents "interpretations" trying to argue its copyrightability based on the idea that time zone data aren't "facts" (apparently not understanding the legal definition). Also the transitions from LMT to the onset of standard time is grossly underreported in the ACS Atlas. This data is nowhere near complete, but it would be nice to keep this info in the tzdb.

From what I understand, the original research by Thomas Shanks et al involved going through local newspapers all over the globe looking for reports of time changes. Many times they would see a time for a baseball game stated and then a later game on another date with a different time format (or something similar), but not exact transition date and time for daylight savings onset, so they probably did an interpolation and preferred weekends for setting this change, but they guard this information as to exactly what criteria they used as a secret. Personally I think the data would have gained a lot more respect if they had cited their sources and found a way to highlight these uncertain periods both in book form and in software.

It would have been nice if the IATASSIM had kept all its data going back to its founding. On Tue, Sep 3, 2013 at 10:47 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:

Brian Inglis wrote:

...

merging zones with different Standard Time start date loses useful historical data

That's not a problem for the proposed change, because in this particular case, no useful historical data are lost. There is no real evidence that the start dates actually differed. All we have is some guesswork from Shanks. The Shanks data often contain guesswork, and the abovementioned transition dates from LMT are almost certainly part of that guesswork.

The associated time zone abbreviations are also guesswork (in this case, my guesswork and not Shanks'), so they do not contain any useful historical data either.

Paul Eggert wrote:

The tz data have long been maintained trying to fix stuff like this when I find it. In several places I've corrected Shanks data when I think it's bogus,

Your recent discussion of Shanks's methodology and quality, and how to deal with it, has been most enlightening. I think it would be useful for you to put these notes into the distributed files, either as a section in Theory or as a new Sources file. -zefram

On 15 September 2013 07:32, Paul Eggert <eggert@cs.ucla.edu> wrote:

+ * The tz database does not record the earliest time for which a + zone's data is thereafter valid for every location in the region. + For example, Europe/London is valid for all locations in the its + region after GMT was made the standard time, but the date of + standardization (1880-08-02) is not in the tz database, other than + in commentary. For many zones the earlist time of validity is + unknown.

"all locations in the its region" is off, but I'm not sure what to change it to (simply "all locations in its region" without the "the"? "all locations in the appropriate region" or the like?). Also, in the last sentence, "earlist" should be "earliest". Cheers, Philip -- Philip Newton <philip.newton@gmail.com>

What a waste of a morning. Changing the motherboard only took 30 minutes, but I'm 3 hours into getting 'vista' running again ... unfortunately need that for the accounts software - and I need to get an end of year run :( Oh for the time when I can use Linux software and still keep the accountant happy :( Paul Eggert wrote:

Good idea. I pushed this patch. This affects only commentary, so it should be safe.

Nice summary of the state of play ... But I feel that the 'problem' of managing what good history is available is still being brushed under the carpet ... hoping it will go away? Filtering the data before building a post-1970 POSIX based 'clock' is fine, and that is what is currently being targeted. One thing that comes to mind though is that currently I'm seeing 'summertime' transitions prior to 1970. Derick has confirmed that he is using the binary data to build the PHP libraries, so presumably these are currently included? What will the situation be post the 'post-1970 only' clean-up? Will these be dropped back to a simple time offset? It's a bit like the problem of identifying which version of tz data is being used so you know that it may be out of date, but there may be two versions of a new distribution? As I have said, I don't use 'seconds' as my time base anyway, so processing leapseconds is just a matter of using 1/86401 on a leapsecond day rather than 1/86400 so I'm also not restricted when it comes to fractional seconds. So it would complete the picture if the rare rules like the Netherlands could be incorporated while not affecting the base POSIX rules? It is almost a case that 'extended mode' should support fractional seconds anyway? Working mainly from database stored material, the concept of 'NULL' as an answer has a number of advantages, but is not practical when using numeric based results. I keep coming back to some sort of 'timezone' result which is in essence 'NULL' such as any LMT results can be taken as 'can be recalculated using local location data' while 'LMTZ' indicates a documented local standard. This removes the need to create any new timezones for what is essentially outside the scope of the 'default' TZ distribution? Given that the theory file contains additional documentation on the content of the 'data' package, would it not make sense to to include it with that package? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

Paul Eggert wrote:

...

This removes the need to create any new

...

timezones for what is essentially outside the scope of the 'default' TZ distribution? I don't think anybody's proposing creating a new zone for every location on the planet, no. The problem is that we'd need thousands of zones merely to handle standard time.

That is the bit I'm not currently getting? I can't see that there are that many more 'standard time' timezones. Yes there may be a lot of cross reference links that indicate when a location joined some sort of standard time, and I'm happy that is a separate list, but why would there be 'thousands' of new timezones? Perhaps it's just interpretation, but if there is evidence for thousands more why are we saying it's all be lost? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

The ACS Atlas would give a clue as to how many would be needed; approximately 2000 zones in the Olson format would be needed to match what is in the ACS atlas and it isn't really "complete". The mapping of time zones appears mathematically to be like a logarithmic spike with one end toward relative uniformity (around 400 zones on the low end after 1970) and the other end to where each clock has its own unique time zone history (aka LMT). It is a matter of where you want to draw the line. The farther in the past you go toward the date of onset of standard time or LMT the more chaotic it gets (to the point where it simply would be impossible to have certainty for many locations). Due to the fact that most people reside in cities, ACS went about as far as reasonably possible given the limited manpower. It was no doubt an extremely tedious task, but I do wish that they had made their research methods and criteria for selecting daylight transitions public. As the data currently stand, my confidence in transition times for DST for dates in the first half of the 20th century is shaky at best. I only feel relatively confident if it is near the dead of winter or the height of summer for this period in history. On Mon, Sep 16, 2013 at 9:32 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:

Lester Caine wrote:

...

why would there be 'thousands' of new timezones?

Because, if one defines a "region" to be "a set of geographical locations where civil-time clocks have agreed since standard time was introduced", there must be thousands of such regions, each unique as far as UT-offset history goes. Shanks identifies over 300 such regions for Indiana alone, and most likely he missed some. (Again, I'm ignoring UT offset before standard time was introduced, and I'm ignoring the date of transition to standard time.)