
Oops, looks like I was mistaken. Howard's library appears to be not (specifically) for Arduino, but a C++ library for Windows (and maybe other) platforms. Maybe it will compile in the Arduino IDE, though I suspect it will be problematic, if possible at all. And I've so far found only one TZdb mirror that keeps a current copy of the data - very disappointing. It's a shame the whole TZdb scheme is so user-unfriendly. Quite discouraging. Regards, Daniel -----Original Message----- From: Daniel Ford [mailto:dfnojunk@gmail.com] Sent: Saturday, November 04, 2017 7:03 PM To: 'Time zone mailing list' Subject: RE: [tz] Beginner's help request Woohoo! I've discovered (and downloaded) an Arduino library, written by Howard E. Hinnant: "This timezone library is a complete parser of the IANA timezone database. It provides for an easy way to access all of the data in this database, using the types from "date.h" and <chrono>. The IANA database also includes data on leap seconds, and this library provides utilities to compute with that information as well." All I need to find now are established TZdb mirrors that keep their copy up-to-date, so I don't overload the master TZdb with queries. Can anyone point me to any, please? Regards, Daniel --------------- Daniel Ford (Gerroa, Australia)

On 11/07/2017 10:38 PM, Daniel Ford wrote:
It's a shame the whole TZdb scheme is so user-unfriendly.
To some extent it's self-defense, as we don't have the resources to be friendly to the billions of potential tzdb users. The main problem here is a resource issue, not a technical issue. Perhaps you could help organize a set of sites willing to volunteer to be tzdb mirrors and to take requests from individual users. As long as the need is small you could mirror the data yourself. Perhaps you could use a free website service like Neocities, for example.

Paul, My experience with 'free' website hosts is not good. They don't remain free for long. If you haven't upgraded (within a few years) to something that costs, they soon change their 'policies' and shut your website down. My own website, currently hosted for a very low monthly sum, is not sufficiently permanent to qualify for long-term TZdb mirroring. And I don't have any contacts in the hosting industry that I could ask to host. I have to say that I'm surprised that an organisation like IANA doesn't have the resources available to properly host the TZdb for 'general access'. No doubt there are hidden reasons, though. Perhaps we need a benign(?) large organisation like Google to take on that free mirroring task? Has anyone spoken with them about it? Regards, Daniel -----Original Message----- From: Paul Eggert [mailto:eggert@cs.ucla.edu] Sent: Thursday, November 09, 2017 6:01 AM To: Daniel Ford; Time Zone Mailing List Subject: Re: [tz] FW: Beginner's help request On 11/07/2017 10:38 PM, Daniel Ford wrote:
It's a shame the whole TZdb scheme is so user-unfriendly.
To some extent it's self-defense, as we don't have the resources to be friendly to the billions of potential tzdb users. The main problem here is a resource issue, not a technical issue. Perhaps you could help organize a set of sites willing to volunteer to be tzdb mirrors and to take requests from individual users. As long as the need is small you could mirror the data yourself. Perhaps you could use a free website service like Neocities, for example.

Daniel Ford wrote:
Has anyone spoken with them about it?
Not as far as I know. If you're interested in long-term support, you might want something more like the Long Now Foundation. http://longnow.org/

LOL! An interesting read, though. Regards, Daniel Daniel Ford wrote:
Has anyone spoken with them about it?
Not as far as I know. If you're interested in long-term support, you might want something more like the Long Now Foundation. http://longnow.org/

It requires a C++11 compiler. There’s one function: current_zone(), that is highly platform-specific. I have no idea if the current implementation (e.g. what works for linux) would work on Arduino. If you’re interested in helping me port to Arduino, I would be interested in accepting patches that made that work. Howard On Nov 7, 2017, at 11:38 PM, Daniel Ford <dfnojunk@gmail.com> wrote:
Oops, looks like I was mistaken. Howard's library appears to be not (specifically) for Arduino, but a C++ library for Windows (and maybe other) platforms. Maybe it will compile in the Arduino IDE, though I suspect it will be problematic, if possible at all.
And I've so far found only one TZdb mirror that keeps a current copy of the data - very disappointing. It's a shame the whole TZdb scheme is so user-unfriendly. Quite discouraging.
Regards, Daniel
-----Original Message----- From: Daniel Ford [mailto:dfnojunk@gmail.com] Sent: Saturday, November 04, 2017 7:03 PM To: 'Time zone mailing list' Subject: RE: [tz] Beginner's help request
Woohoo! I've discovered (and downloaded) an Arduino library, written by Howard E. Hinnant:
"This timezone library is a complete parser of the IANA timezone database. It provides for an easy way to access all of the data in this database, using the types from "date.h" and <chrono>. The IANA database also includes data on leap seconds, and this library provides utilities to compute with that information as well."
All I need to find now are established TZdb mirrors that keep their copy up-to-date, so I don't overload the master TZdb with queries. Can anyone point me to any, please?
Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) <README.txt>

Howard, Thanks for your response to my comments. And I tip my hat to you for the great effort in developing a library that handles the convoluted TZdb! I have to mention at the outset that: 1. I'm a raw novice at the whole Arduino scene (and see further comment below); 2. I have zero knowledge and experience of C++. My background is a professional electronics engineer, having spent the past 30 or so years designing/developing embedded systems, both hardware and firmware. My firmware experience (self-taught programmer) is mainly with assembler (various devices) and C. That said, I'm always willing to learn new skills in my retirement (helps stave off dementia, they say! :-) I hate Arduino! It's a platform that seems to have been developed by, and is now largely supported by, hobbyists. Detailed technical documentation is invariably somewhere between atrocious and non-existent. Anathema to a professional engineer! So why am I using it? Where else will you buy a computing platform with an in-built wi-fi module (both client and AP modes, even simultaneously!) for under AUD5 including postage (<US4)? [There could be similar Raspberry Pi bargains, but they would likely come with similar issues.] Anyway, I was happy to read (on Stackoverflow): "As of version 1.6.6, the Arduino IDE enables C++11 by default." So, in-between reading a C++ tutorial, what I guess is my next step is to 'install' your library in the Arduino environment, then find or write a simple program to call on, for example, current_zone(), and see what errors are thrown-up. I'm very grateful for your offer to assist. Be prepared for lots of 'dumb' beginner questions! (I'll communicate with you directly rather than clutter the TZ list with this side-exercise. But if you eventually come up with an Arduino library version, we should definitely announce it on the TZ discussion list.) Regards, Daniel

On 2017-11-09 04:34, Daniel Ford wrote:
Thanks for your response to my comments. And I tip my hat to you for the great effort in developing a library that handles the convoluted TZdb!> I have to mention at the outset that: 1. I'm a raw novice at the whole Arduino scene (and see further comment below); 2. I have zero knowledge and experience of C++. My background is a professional electronics engineer, having spent the past 30 or so years designing/developing embedded systems, both hardware and firmware. My firmware experience (self-taught programmer) is mainly with assembler (various devices) and C. That said, I'm always willing to learn new skills in my retirement (helps stave off dementia, they say! :-) I hate Arduino! It's a platform that seems to have been developed by, and is now largely supported by, hobbyists. Detailed technical documentation is invariably somewhere between atrocious and non-existent. Anathema to a professional engineer! So why am I using it? Where else will you buy a computing platform with an in-built wi-fi module (both client and AP modes, even simultaneously!) for under AUD5 including postage (<US4)? [There could be similar Raspberry Pi bargains, but they would likely come with similar issues.]
RPi comes with quad 32 bit processors, 1GB memory, enet, BT, wifi, serial, supports up to 32GB SDHC, USB flash, or USB magnetic drives, a fairly standard embedded Debian Linux distro, thousands of utilities and packages, and regular updates to tzdata and all the other packages. You could easily host your own mirror on one. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

Howard, Thank you very much for trying to help with porting your library to Arduino. But after your recent reply to my trial compilation errors and your suggestions for further steps, I have decided that I would need to be reasonably fluent in C++, plus make major changes to your code, before I would have any hope of getting it to work in the Arduino IDE. The amount of time required for those tasks would almost certainly exceed the time taken to refine my earlier-proposed strategy of parsing the TZdb text files to ascertain the offset and DST rules for a specified locale, so I'm sorry to say I'm abandoning my attempt to port your library to Arduino. I've also decided, in the absence of a significant number of TZdb mirror sites, to host the unzipped files on my own server, and maybe some friends' servers too. Hopefully at least some of those will be around for the lifetime of my clocks! Regards, Daniel -----Original Message----- From: Howard Hinnant [mailto:howard.hinnant@gmail.com] Sent: Thursday, November 09, 2017 7:18 AM To: Daniel Ford Cc: tz@iana.org Subject: Re: [tz] FW: Beginner's help request It requires a C++11 compiler. There’s one function: current_zone(), that is highly platform-specific. I have no idea if the current implementation (e.g. what works for linux) would work on Arduino. If you’re interested in helping me port to Arduino, I would be interested in accepting patches that made that work. Howard On Nov 7, 2017, at 11:38 PM, Daniel Ford <dfnojunk@gmail.com> wrote:
Oops, looks like I was mistaken. Howard's library appears to be not (specifically) for Arduino, but a C++ library for Windows (and maybe other) platforms. Maybe it will compile in the Arduino IDE, though I suspect it will be problematic, if possible at all.
And I've so far found only one TZdb mirror that keeps a current copy of the data - very disappointing. It's a shame the whole TZdb scheme is so user-unfriendly. Quite discouraging.
Regards, Daniel
-----Original Message----- From: Daniel Ford [mailto:dfnojunk@gmail.com] Sent: Saturday, November 04, 2017 7:03 PM To: 'Time zone mailing list' Subject: RE: [tz] Beginner's help request
Woohoo! I've discovered (and downloaded) an Arduino library, written by Howard E. Hinnant:
"This timezone library is a complete parser of the IANA timezone database. It provides for an easy way to access all of the data in this database, using the types from "date.h" and <chrono>. The IANA database also includes data on leap seconds, and this library provides utilities to compute with that information as well."
All I need to find now are established TZdb mirrors that keep their copy up-to-date, so I don't overload the master TZdb with queries. Can anyone point me to any, please?
Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) <README.txt>

Daniel Ford wrote:
I've also decided, in the absence of a significant number of TZdb mirror sites, to host the unzipped files on my own server,
If you're doing that, you don't need the Arduino to process the files of the tzdb distro directly. You could process the files on your server into a more Arduino-friendly format. Host compiled tzfiles, as an initial version. For extra credit you could cut down the tzfiles to cover only the restricted time span of interest to the clocks (ten minutes ago to ten years hence). Your downloads could be really short in that case. In fact, small enough to look at UDP transport, for bonus network lightness. DNS is also a viable transport mechanism for objects that small. -zefram

Thanks Zefram and Paul for your ongoing assistance. I managed to find versions of zic.exe and zdump.exe that run in my Windoze environment (using GygWin64), and have played with them a little to see what they do (the documentation is a bit sparse). Zic produces binary data that looks like it will be of little use to me (since I don't know what it is!), but zdump produces a potentially useful ASCII output. Given that there's no simple way of finding the source file for a given region/city, I guess the logical thing to do is concatenate all the source files into one, and search that for the desired region/city. So I delved into Makefile and extracted the line that specifies the source files (PRIMARY_YDATA= africa antarctica asia australasia europe northamerica southamerica) and concatenated those files. I presume this set of files is unlikely to change in the foreseeable future? But zdump produces DST output in 'absolute date' form rather than the 'current rules' I'm looking for. The Arduino code (by Jack Christensen) I'm planning to use (since it's so well-written and does all the timekeeping and DST computation I need) runs on 'rules' (as indicated in my first post on 14-Oct-2017), not absolute dates, so I still need something that extracts current rules from TZdb, not absolute change date/times. But I think I can now extract those rules from the concatenated source files, using a parsing method much as I described in my 31-Oct-2017 post, modified as per Zefram's comments on 1-Nov (maybe 31-Oct in his timezone). And as Zefram said, most of this pre-processing can be done on my server. So a clock would simply send a query to my server (here's an equivalent human interaction!): "Have my rules changed in the last week?", and the server would reply either: "No, keep using your current rules" or "Yes, here are your new rules." Another issue to be decided is how my server checks your server, perhaps weekly, for updates. Paul mentioned 'get the Last-Modified: header', but I'm not sure what that is and how my server would get it. Ideally there's a way for a server to find out whether the TZdb has been updated without having to download files first. Regards, Daniel

Daniel Ford wrote:
Paul mentioned 'get the Last-Modified: header', but I'm not sure what that is and how my server would get it.
That's standardized here: https://tools.ietf.org/html/rfc7232#section-2.2 For example, on my Fedora 27 host, I can run the following two shell commands: wget -S https://www.iana.org/time-zones/repository/tzdb/tzdata.zi wget -N -S https://www.iana.org/time-zones/repository/tzdb/tzdata.zi and observe the attached transcript. The "Last-Modified:" header is one of the HTTP headers I get back from the first "wget". The second "wget" uses a conditional get, and gets a "304 Not Modified" response which means the local copy of tzdata.zi is already up-to-date, and need not be refetched. Say, it looks like the data.iana.org redirects are working! That's new. And these servers support the "Etag:" header, which is better than "Last-Modified:" in some ways. So it looks like you should be using "Etag:" instead of "Last-Modified:", as described here: https://tools.ietf.org/html/rfc7232#section-2.3

Daniel Ford wrote:
Zic produces binary data that looks like it will be of little use to me (since I don't know what it is!),
That's the binary tzfile format, documented in tzfile.5{,.txt}. It is an excellent format for your purposes.
zdump produces a potentially useful ASCII output.
The best way to use zdump is to compare its output against the output of your own code, to make sure you're parsing tzfiles correctly.
into Makefile and extracted the line that specifies the source files (PRIMARY_YDATA=
That's not the full set of data files, though it might be a subset that you want to work with, because it covers the distinct geographical zones. The full canonical set of zones, including non-geographical zones and links, comes from the files specified by TDATA.
I presume this set of files is unlikely to change in the foreseeable future?
In practice it changes very slowly. There are no guarantees, though. Nor are there guarantees about most of the Makefile macro names.
But zdump produces DST output in 'absolute date' form rather than the 'current rules' I'm looking for.
It's unrealistic to expect rules. Quite often there is no set of rules that will apply correctly to each of the next few years, because rule changes have been scheduled. In the general case you *must* cope with having different rules for different years, in the ultimate case separate rules for each year, so it should be no hardship to *always* handle upcoming behaviour in the form of explicit transitions. The binary tzfile usually provides a ruleset applicable to indefinitely many years, in the form of a POSIX TZ string. But that's not the current rules, it's the rules to be applied when all of the file's explicit transitions have run out. -zefram

On 11/22/2017 02:30 AM, Zefram wrote:
The binary tzfile usually provides a ruleset applicable to indefinitely many years, in the form of a POSIX TZ string. But that's not the current rules, it's the rules to be applied when all of the file's explicit transitions have run out.
Changing the subject (and subject line).... Currently zic generates a bunch of explicit transitions at the end of the time transition table, even if these transitions merely repeat what a TZ string would generate. This is to support older clients that grok only version-1 tzfile format, and which don't know about the TZ string (which was introduced in version 2 tzfile format). Since version 2 format was introduced in release 2006b, I'm thinking now may be a good time to add an option to zic to suppress explicit transitions at the end of the table if these transitions agree with the TZ string. This will help uncover buggy client code that mishandles TZ string values and therefore mishandles future timestamps. By default this option will be off now (meaning no change to the default behavior), but we could turn it on later.

On 11/22/2017 01:49 PM, Paul Eggert wrote:
On 11/22/2017 02:30 AM, Zefram wrote:
The binary tzfile usually provides a ruleset applicable to indefinitely many years, in the form of a POSIX TZ string. But that's not the current rules, it's the rules to be applied when all of the file's explicit transitions have run out.
Changing the subject (and subject line)....
Currently zic generates a bunch of explicit transitions at the end of the time transition table, even if these transitions merely repeat what a TZ string would generate. This is to support older clients that grok only version-1 tzfile format, and which don't know about the TZ string (which was introduced in version 2 tzfile format). Since version 2 format was introduced in release 2006b, I'm thinking now may be a good time to add an option to zic to suppress explicit transitions at the end of the table if these transitions agree with the TZ string. This will help uncover buggy client code that mishandles TZ string values and therefore mishandles future timestamps. By default this option will be off now (meaning no change to the default behavior), but we could turn it on later.
This option makes sense to me. Would it also be possible to have this option (or a separate one) create the first header and body (32-bit time values) with counts of zero? All of the data that would appear in that body is duplicated in the second body anyways. -- Kenneth Murchison Cyrus Development Team FastMail Pty Ltd

Ken Murchison wrote:
Would it also be possible to have this option (or a separate one) create the first header and body (32-bit time values) with counts of zero?
That'd be a change to the format, since the intent (as I understand it) is for the 32- and 64-bit parts to agree for timestamps in 32-bit range. (Although this isn't written down; it should be.) So this should be a separate zic option. I expect 32-bit clients typically look only at the 32-bit part and would misfire if it were omitted, so if we added a zic option to generate such a format it would surely not be the default at first. Still, it would make sense to add the option, as 32-bit clients will become obsolete in a couple of decades anyway, and we might as well get the ball rolling.

Paul Eggert <eggert@cs.ucla.edu> writes:
Ken Murchison wrote:
Would it also be possible to have this option (or a separate one) create the first header and body (32-bit time values) with counts of zero?
That'd be a change to the format, since the intent (as I understand it) is for the 32- and 64-bit parts to agree for timestamps in 32-bit range. (Although this isn't written down; it should be.) So this should be a separate zic option.
If part of the objective is to expose code that fails to deal with the newer format options, I think you need to be able to generate files in which (a) the 32-bit data isn't there, *and* (b) the TZ string actually controls results for current timestamps in some popular zones. If the TZ string only determines post-2038 results in Outer Mongolia, it's going to take a long time for anyone to notice if their tzdb client code is broken. I don't know if Paul's proposed behavior change would make (b) true automatically or not. regards, tom lane

Tom Lane wrote:
I think you need to be able to generate files in which (a) the 32-bit data isn't there,*and* (b) the TZ string actually controls results for current timestamps in some popular zones.
Yes, (b) is part of my proposal. Sorry I wasn't clear. Because the tzfile format is independent of the current time and date, (b) follows the format's rules and should work -- though you're right, some client-code bugs may be exposed by (b). (a), on the other hand, is not clearly contemplated by tzfile(5), and may well break non-buggy code in 32-bit clients that never look at the 64-bit data. So it requires a change to the tzfile format and is a bigger deal.

Thanks Paul and Zefram (and Brian via PM). I really do appreciate all the time you're spending on educating me about the TZdb and its use. I'm almost reaching the state of 'data overload'! :-) I have now contacted my website hosting service to ask them about what I can and cannot do on their server, and the brief answer was (a) CAN: "create a shell script to manipulate the data and create new files under your hosting service on the server", and (b) CANNOT: "its possible to run .exe on the Linux server but some additional package installation needed. However, we can't install this package on the server as your website is hosted in the shared server environment." So that rules out running zic, zdump and any such programs. All I can do is run shell commands. Fortunately the typical Linux shell seems to be pretty competent, including wget, unzip/untar commands, and this might be enough to have my TZdb mirror running autonomously without me having to do anything (after setting it up with a script, of course). So what I envisage my server doing, perhaps weekly (using Cronjob), is: 1. Check if tzdata-latest.tar.gz has changed, and download it if so; 2. If a new version was downloaded, unpack it to the individual files into my TZdb web page directory; 3. Copy the rule files to a single file: 'cp africa antarctica asia australasia europe northamerica southamerica zoneData.txt' (Yes, I understand that's not the full set of data files, but it is the subset that I need for my purpose, as Zefram inferred.) So now comes my next beginner's question (not being a Linux user), concerning steps 1 and 2 above. Looking through the wget manual, this method appears to show promise (thanks to Brian for this suggestions): wget -N ftp://ftp.iana.org/tz/tzdata-latest.tar.gz But what is not clear in the manual is how I check if a newer file has been downloaded. They talk about the error codes returned by wget, but without explaining explicitly what is returned when wget (a) finds the files the same and doesn't download anything, or (b) finds a newer version and downloads it. If these situations produce different error codes, I can use an 'if' statement in my script to process the new data iff a new file arrives. Not wanting to load too many questions into one mail (nor overload my addled brain!), I'll leave till later a summary of how I propose to handle the clock-end firmware interaction with my 'TZdb server'. Regards, Daniel

Daniel Ford wrote:
So that rules out running zic, zdump and any such programs.
It's a bit unclear, but I think compiling zic et al from tzdb and running them locally falls under the rubric of "shell script to manipulate the data and create new files". You don't need zic to be installed as a proper part of the shared server environment. What you do need installed (which should be unobjectionable if they're not already installed) are make(1) and a C compiler.
3. Copy the rule files to a single file: 'cp africa antarctica asia australasia europe northamerica southamerica zoneData.txt'
That's not how cp(1) is used. You'll want cat(1) to concatenate them. -zefram

Zefram, Yes, it is unclear whether your suggestion about executables would work, but I'll probably not try. My provider gives good support despite the pittance I pay them, and I don't want to upset them. I'm assuming their reasoning is that allowing users to run 'unknown' executables could potentially crash their server, disadvantaging unrelated users sharing that server, and I think that's a reasonable restriction in the circumstances. Thanks for the correction regarding cp versus cat (showing my Linux ignorance). I misunderstood one of the Linux references I read, where it showed multiple files being copied with cp. I now understand that simply copies all the source files *individually* to a destination directory, not concatenating them. So 'cat' it is! Regards, Daniel
participants (7)
-
Brian Inglis
-
Daniel Ford
-
Howard Hinnant
-
Ken Murchison
-
Paul Eggert
-
Tom Lane
-
Zefram