I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable. Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information... (I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.) I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC. The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches. For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow. Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information?? I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing. Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as... // Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld! ... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler! Any help in using the TZ database will be much appreciated (and acknowledged in my source code). Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) "My loathings are simple: stupidity, oppression, crime, cruelty, soft music." - Vladimir Nabokov
Daniel, Just a short message to help you get quickly started along the right path. A guide to interpreting the source data can be found in tz-how-to.html; I'm not sure offhand whether that's in the particular tar file you have, but a development version is available from https://github.com/eggert/tz/b lob/master/tz-how-to.html For your idea of a text-only table of already-parsed data, you may consider the zdump utility that is packaged in the distribution. For your example, to obtain the offsets for Australia/Hobart for 2017 only: $ zdump -vc 2017,2018 Australia/Hobart Australia/Hobart -9223372036854775808 = NULL Australia/Hobart -9223372036854689408 = NULL Australia/Hobart Sat Apr 1 15:59:59 2017 UTC = Sun Apr 2 02:59:59 2017 AEDT isdst=1 gmtoff=39600 Australia/Hobart Sat Apr 1 16:00:00 2017 UTC = Sun Apr 2 02:00:00 2017 AEST isdst=0 gmtoff=36000 Australia/Hobart Sat Sep 30 15:59:59 2017 UTC = Sun Oct 1 01:59:59 2017 AEST isdst=0 gmtoff=36000 Australia/Hobart Sat Sep 30 16:00:00 2017 UTC = Sun Oct 1 03:00:00 2017 AEDT isdst=1 gmtoff=39600 Australia/Hobart 9223372036854689407 = NULL Australia/Hobart 9223372036854775807 = NULL -- Tim Parenti sent from my Android phone On 14 Oct 2017 19:15, "Daniel Ford" <dfnojunk@gmail.com> wrote:
I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable.
Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information...
(I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.)
I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC.
The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches.
For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow.
Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information??
I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing.
Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as...
// Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld!
... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler!
Any help in using the TZ database will be much appreciated (and acknowledged in my source code).
Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) "My loathings are simple: stupidity, oppression, crime, cruelty, soft music." - Vladimir Nabokov
Tim Parenti wrote:
For your idea of a text-only table of already-parsed data, you may consider the zdump utility that is packaged in the distribution.
It sounds like what's wanted is more a table like this: Africa/Abidjan GMT0 Africa/Accra GMT0 Africa/Algiers CET-1 Africa/Bissau GMT0 Africa/Cairo EET-2 Africa/Casablanca WET0WEST,M3.5.0,M10.5.0/3 Africa/Ceuta CET-1CEST,M3.5.0,M10.5.0/3 ... where the first column is the TZ name, and the second is the prediction of the future behavior of clocks in the corresponding location, in nearly-POSIX TZ string format. We don't distribute a table like that now, though I suppose we could generate one automatically from the information that is already there.
...the first column is the TZ name, and the second is the prediction of the future behavior of clocks in the corresponding location, in nearly-POSIX TZ string format. We don't distribute a table like that now, though I suppose we could generate one automatically from the information that is already there.
Yes; all that's need is a bit of elaboration on cd /YOUR/SYSTEM/PREAMBLE/zoneinfo tail -n 1 `find * ! -name "*.tab"` --ado On Sat, Oct 14, 2017 at 10:33 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
Tim Parenti wrote:
For your idea of a text-only table of already-parsed data, you may consider the zdump utility that is packaged in the distribution.
It sounds like what's wanted is more a table like this:
Africa/Abidjan GMT0 Africa/Accra GMT0 Africa/Algiers CET-1 Africa/Bissau GMT0 Africa/Cairo EET-2 Africa/Casablanca WET0WEST,M3.5.0,M10.5.0/3 Africa/Ceuta CET-1CEST,M3.5.0,M10.5.0/3 ...
where the first column is the TZ name, and the second is the prediction of the future behavior of clocks in the corresponding location, in nearly-POSIX TZ string format. We don't distribute a table like that now, though I suppose we could generate one automatically from the information that is already there.
Date: Sat, 14 Oct 2017 23:45:20 +1100 From: "Daniel Ford" <dfnojunk@gmail.com> Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAHqgwV9xKnNLs4iwQOaaXEzCgAAAEAAAAOYZ/9puZUZPkkKsORAOty0BAAAAAA==@gmail.com> The issue of how to get the current offset, and the next planned alteration (if any) (and perhaps one or two more after that) is, or has been, dealt with by others already - though I might add, that the tzdata files that you looked at are really only intended to be processed by zic, though there are a couple of other tools around that also do it. The output from zic however is a binary file that contains almost exactly what you want - the suggestion to use zdump is basically just converting some of that binary data into text form, but really, for what you want, if you get text, you're just going to have to translate it back to binary again anyway, so you might as well just parse the zoneinfo binary files, and get the data from them (that's what the unix libc functions do when some application wants to convert a "seconds since..." timestamp into a clock (time of day, plus date) type time. There is (at least one) windows port of the tzcode I believe, so doing it this way, even if you have to suffer doing it on windows, is not impossible. But ... | Given that DST 'rules' can change at the whim of governments, my firmware | will need to annually look up 'current' DST rules from the web. This is actually the hard part. Annually is not nearly often enough, the rules can change with just weeks (or sometimes) days, of advance notice. If your clocks are using old data, they'll end up showing the wrong time if they are in an affected area. Either you need to have them poll daily (at worst) which could be a huge load on whatever server you have them contact (if your clock becomes successful, of course) or you will need to find some other way to notify the clocks when they ought to be fetching updates. If you are polling you could lessen the load a bit by not using HTTP (or anything related to "the web") and instead using a simpler protocol, which could start with a simple UDP type "has anything changed for zone xxxx since time X" which would send a simple "last change for zone xxxx was @Y" or something reply (one small simple packet each way.) Then only when there has been a change would you use a more robust protocol to actually fetch the updated data (that could be http based, if that is essential). If a new protocol for the UDP part sounds too risky, you could even (ab)use the DNS - have the clock send a DNS query for a TXT record for zone-name.some-domain.net (were some-domain.net is something registered just for the purpose). The returned TXT record would contain the data that you need. That way you get the benefits of DNS caching. The point is that this part needs careful consideration and design, as whatever you start with will be very hard to ever change once you have devices out in the wild, and that you really do need to verify the data frequently to have any confidence in it at all -- and someone needs to make sure the server is being updated - monitoring this list probably, as it happens sometimes that zones are updated with so little notice that the tzdata distributions cannot keep up, and only include the updated data sometime after the change has become effective - usually this list will hear about it very quickly, which would allow someone monitoring what appears here to update the server even before there's new tzdata available. kre ps: I would suggest including a GPS receiver in the clock - as they are in every phone these days, they have become cheap enough for this I think. That way your clock can display accurate time with much less network overhead than it would using NTP (and typically with much less code.) The network time option would still be useful for clocks located in environments where satellite reception is not possible though.
Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments).
If you allow the user to enter the City Name, please note that the user might enter "Hovarta", which is the (made-up) localised version of the name of the city. cf Kolkotta, Wien, etc This issue turns up regularly on the list (along with "But that is not the biggest city, including suburbs). The TZ database does not deal with UI issues, in this respect, you will have to handle them. Will your clock have a UI? Accessible as a web page, or with up/down/select buttons? -- Sanjeev Gupta +65 98551208 http://www.linkedin.com/in/ghane On Sat, Oct 14, 2017 at 8:45 PM, Daniel Ford <dfnojunk@gmail.com> wrote:
I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable.
Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information...
(I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.)
I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC.
The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches.
For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow.
Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information??
I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing.
Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as...
// Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld!
... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler!
Any help in using the TZ database will be much appreciated (and acknowledged in my source code).
Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) "My loathings are simple: stupidity, oppression, crime, cruelty, soft music." - Vladimir Nabokov
Sanjeev, Thanks for your comments, which I'll address before the other feedback I've received, since the others will require more considered responses... Firmware planning has barely started, but my current intention is to have the NodeMCU start-up (first-time only) in Wi-Fi AP mode, with a specified SSID/password. So the user will be expected to log onto that AP to work through the set-up pages. Apart from the user's own SSID/Password (for their own 'home' AP), no free-form text will be allowed. They must select first their 'region', from a displayed list, and then their 'reference city' from another displayed list (which, of course, will only contain cities from the *nominated* region). So naturally I'll choose those region and city names to be compatible with those used in the TZdb. Regards, Daniel ________________________________ From: Sanjeev Gupta [mailto:ghane0@gmail.com] Sent: Sunday, October 15, 2017 3:24 PM To: Daniel Ford Cc: tz@iana.org Subject: Re: [tz] Beginner's help request
Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments).
If you allow the user to enter the City Name, please note that the user might enter "Hovarta", which is the (made-up) localised version of the name of the city. cf Kolkotta, Wien, etc This issue turns up regularly on the list (along with "But that is not the biggest city, including suburbs). The TZ database does not deal with UI issues, in this respect, you will have to handle them. Will your clock have a UI? Accessible as a web page, or with up/down/select buttons? -- Sanjeev Gupta +65 98551208 http://www.linkedin.com/in/ghane On Sat, Oct 14, 2017 at 8:45 PM, Daniel Ford <dfnojunk@gmail.com> wrote: I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable. Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information... (I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.) I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC. The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches. For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow. Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information?? I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing. Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as... // Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld! ... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler! Any help in using the TZ database will be much appreciated (and acknowledged in my source code). Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) "My loathings are simple: stupidity, oppression, crime, cruelty, soft music." - Vladimir Nabokov
Tim, Thanks for these pointers to the 'how-to' and zdump.c. I'll read the how-to when I get a free moment, and maybe also try to compile zdump.c for MS-DOS, to see how it works. I fear it might not translate easily to an Arduino environment, though! The other issue I have with the results you've listed is that they are using absolute dates rather than quoting the official rules (e.g. 'first Sunday in October', 'first Sunday in April', etc). This would be fine as long as my clock has access to the TZdb each year, but what if the database someday ceases to exist (or be maintained)? Unlikely though that is, I would nevertheless like to be able to, in that unthinkable circumstance, continue adjusting DST for the user's locale, based on the actual rules and calculating my own dates each year. Regards, Daniel ________________________________ From: Tim Parenti [mailto:tim@timtimeonline.com] Sent: Sunday, October 15, 2017 11:22 AM To: Daniel Ford Cc: Time Zone Mailing List Subject: Re: [tz] Beginner's help request Daniel, Just a short message to help you get quickly started along the right path. A guide to interpreting the source data can be found in tz-how-to.html; I'm not sure offhand whether that's in the particular tar file you have, but a development version is available from https://github.com/eggert/tz/blob/master/tz-how-to.html <https://github.com/eggert/tz/blob/master/tz-how-to.html> For your idea of a text-only table of already-parsed data, you may consider the zdump utility that is packaged in the distribution. For your example, to obtain the offsets for Australia/Hobart for 2017 only: $ zdump -vc 2017,2018 Australia/Hobart Australia/Hobart -9223372036854775808 = NULL Australia/Hobart -9223372036854689408 = NULL Australia/Hobart Sat Apr 1 15:59:59 2017 UTC = Sun Apr 2 02:59:59 2017 AEDT isdst=1 gmtoff=39600 Australia/Hobart Sat Apr 1 16:00:00 2017 UTC = Sun Apr 2 02:00:00 2017 AEST isdst=0 gmtoff=36000 Australia/Hobart Sat Sep 30 15:59:59 2017 UTC = Sun Oct 1 01:59:59 2017 AEST isdst=0 gmtoff=36000 Australia/Hobart Sat Sep 30 16:00:00 2017 UTC = Sun Oct 1 03:00:00 2017 AEDT isdst=1 gmtoff=39600 Australia/Hobart 9223372036854689407 = NULL Australia/Hobart 9223372036854775807 = NULL -- Tim Parenti sent from my Android phone On 14 Oct 2017 19:15, "Daniel Ford" <dfnojunk@gmail.com> wrote: I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable. Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information... (I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.) I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC. The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches. For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow. Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information?? I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing. Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as... // Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld! ... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler! Any help in using the TZ database will be much appreciated (and acknowledged in my source code). Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) "My loathings are simple: stupidity, oppression, crime, cruelty, soft music." - Vladimir Nabokov
Paul, Thanks for this, but I actually liked Tim's table better! :-) Perhaps if I can find somewhere that explains 'POSIX TZ string format', I might begin to understand your table. I'll have a look around. PS: I was (before retirement) an embedded systems designer (h/ware + f/ware), with negligible experience in DOS and Windows programming and databases. DBase IV and CA-Clipper (both in MS-DOS) are as close as I've been to database programming, and even that was adapting someone else's code, not originating it! Regards, Daniel -----Original Message----- From: Paul Eggert [mailto:eggert@cs.ucla.edu] Sent: Sunday, October 15, 2017 1:34 PM To: Tim Parenti; Daniel Ford Cc: Time Zone Mailing List Subject: Re: [tz] Beginner's help request Tim Parenti wrote:
For your idea of a text-only table of already-parsed data, you may consider the zdump utility that is packaged in the distribution.
It sounds like what's wanted is more a table like this: Africa/Abidjan GMT0 Africa/Accra GMT0 Africa/Algiers CET-1 Africa/Bissau GMT0 Africa/Cairo EET-2 Africa/Casablanca WET0WEST,M3.5.0,M10.5.0/3 Africa/Ceuta CET-1CEST,M3.5.0,M10.5.0/3 ... where the first column is the TZ name, and the second is the prediction of the future behavior of clocks in the corresponding location, in nearly-POSIX TZ string format. We don't distribute a table like that now, though I suppose we could generate one automatically from the information that is already there.
Arthur, I'm assuming this was directed more to Paul than to me, but many thanks for your input to the discussion. Regards, Daniel ________________________________ From: Arthur David Olson [mailto:arthurdavidolson@gmail.com] Sent: Sunday, October 15, 2017 1:41 PM Cc: Daniel Ford; Time Zone Mailing List Subject: Re: [tz] Beginner's help request
...the first column is the TZ name, and the second is the prediction of the future behavior of clocks in the corresponding location, in nearly-POSIX TZ string format. We don't distribute a table like that now, though I suppose we could generate one automatically from the information that is already there.
Yes; all that's need is a bit of elaboration on cd /YOUR/SYSTEM/PREAMBLE/zoneinfo tail -n 1 `find * ! -name "*.tab"` --ado On Sat, Oct 14, 2017 at 10:33 PM, Paul Eggert <eggert@cs.ucla.edu> wrote: Tim Parenti wrote: For your idea of a text-only table of already-parsed data, you may consider the zdump utility that is packaged in the distribution. It sounds like what's wanted is more a table like this: Africa/Abidjan GMT0 Africa/Accra GMT0 Africa/Algiers CET-1 Africa/Bissau GMT0 Africa/Cairo EET-2 Africa/Casablanca WET0WEST,M3.5.0,M10.5.0/3 Africa/Ceuta CET-1CEST,M3.5.0,M10.5.0/3 ... where the first column is the TZ name, and the second is the prediction of the future behavior of clocks in the corresponding location, in nearly-POSIX TZ string format. We don't distribute a table like that now, though I suppose we could generate one automatically from the information that is already there.
Daniel Ford wrote:
Perhaps if I can find somewhere that explains 'POSIX TZ string format', I might begin to understand your table. I'll have a look around.
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html (Look for "TZ".) This explains the POSIX TZ variable's format. http://man7.org/linux/man-pages/man5/tzfile.5.html (Look for "Version 3 format".) This explains the minor extensions used in tzdb binary files.
Paul, Excellent! Thanks for those references, which I've added to my reading list for the coming week. Regards, Daniel -----Original Message----- From: Paul Eggert [mailto:eggert@cs.ucla.edu] Sent: Sunday, October 15, 2017 5:14 PM To: Daniel Ford Cc: 'Time Zone Mailing List' Subject: Re: [tz] Beginner's help request Daniel Ford wrote:
Perhaps if I can find somewhere that explains 'POSIX TZ string format', I might begin to understand your table. I'll have a look around.
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html (Look for "TZ".) This explains the POSIX TZ variable's format. http://man7.org/linux/man-pages/man5/tzfile.5.html (Look for "Version 3 format".) This explains the minor extensions used in tzdb binary files.
While the tzdist protocol is http and does not (yet) handle zoneinfo format it does seem a long way towards what you're talking about. https://tools.ietf.org/html/rfc7808 The main impediment (I believe) to handling zoneinfo data is coming up with a standard format in an rfc On 10/14/17 23:55, Robert Elz wrote:
Date: Sat, 14 Oct 2017 23:45:20 +1100 From: "Daniel Ford" <dfnojunk@gmail.com> Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAHqgwV9xKnNLs4iwQOaaXEzCgAAAEAAAAOYZ/9puZUZPkkKsORAOty0BAAAAAA==@gmail.com>
The issue of how to get the current offset, and the next planned alteration (if any) (and perhaps one or two more after that) is, or has been, dealt with by others already - though I might add, that the tzdata files that you looked at are really only intended to be processed by zic, though there are a couple of other tools around that also do it.
The output from zic however is a binary file that contains almost exactly what you want - the suggestion to use zdump is basically just converting some of that binary data into text form, but really, for what you want, if you get text, you're just going to have to translate it back to binary again anyway, so you might as well just parse the zoneinfo binary files, and get the data from them (that's what the unix libc functions do when some application wants to convert a "seconds since..." timestamp into a clock (time of day, plus date) type time.
There is (at least one) windows port of the tzcode I believe, so doing it this way, even if you have to suffer doing it on windows, is not impossible.
But ...
| Given that DST 'rules' can change at the whim of governments, my firmware | will need to annually look up 'current' DST rules from the web.
This is actually the hard part. Annually is not nearly often enough, the rules can change with just weeks (or sometimes) days, of advance notice. If your clocks are using old data, they'll end up showing the wrong time if they are in an affected area.
Either you need to have them poll daily (at worst) which could be a huge load on whatever server you have them contact (if your clock becomes successful, of course) or you will need to find some other way to notify the clocks when they ought to be fetching updates. If you are polling you could lessen the load a bit by not using HTTP (or anything related to "the web") and instead using a simpler protocol, which could start with a simple UDP type "has anything changed for zone xxxx since time X" which would send a simple "last change for zone xxxx was @Y" or something reply (one small simple packet each way.)
Then only when there has been a change would you use a more robust protocol to actually fetch the updated data (that could be http based, if that is essential).
If a new protocol for the UDP part sounds too risky, you could even (ab)use the DNS - have the clock send a DNS query for a TXT record for zone-name.some-domain.net (were some-domain.net is something registered just for the purpose). The returned TXT record would contain the data that you need. That way you get the benefits of DNS caching.
The point is that this part needs careful consideration and design, as whatever you start with will be very hard to ever change once you have devices out in the wild, and that you really do need to verify the data frequently to have any confidence in it at all -- and someone needs to make sure the server is being updated - monitoring this list probably, as it happens sometimes that zones are updated with so little notice that the tzdata distributions cannot keep up, and only include the updated data sometime after the change has become effective - usually this list will hear about it very quickly, which would allow someone monitoring what appears here to update the server even before there's new tzdata available.
kre
ps: I would suggest including a GPS receiver in the clock - as they are in every phone these days, they have become cheap enough for this I think. That way your clock can display accurate time with much less network overhead than it would using NTP (and typically with much less code.) The network time option would still be useful for clocks located in environments where satellite reception is not possible though.
On 2017-10-15 00:13, Paul Eggert wrote:
Daniel Ford wrote:
Perhaps if I can find somewhere that explains 'POSIX TZ string format', I might begin to understand your table. I'll have a look around. http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html (Look for "TZ".) This explains the POSIX TZ variable's format. http://man7.org/linux/man-pages/man5/tzfile.5.html (Look for "Version 3 format".) This explains the minor extensions used in tzdb binary files.
The attached shell script will generate output similar to Paul's suggestion, when run against installed binary zoneinfo files. It was tested on Windows Subsystem for Linux/"Bash for Windows"/Ubuntu tzdata 2016j and Cygwin tzdata 2017b. The outputs from running the script against those systems are attached. Either or both systems can be installed under Windows to support Unix scripts and development, and both provide a tzdata package to install the zoneinfo files as part of their base systems. The script was simplified from a long one-liner pipeline using various commands, to do selection of files only using find, strings to extract text from each binary file, sed to pick the last line containing the POSIX rule, skip .../posix/..., .../right/..., *?.?* files, and strip the leading directory path to leave the zone paths. Storage space required could be reduced by filtering the list to only contain entries in zone1970.tab, front end compression of zone paths, and back end compression of common/duplicate rules. This could be done by a program to generate source code for compact C or asm tables to build into your system, using the zones and POSIX rules in the list, or the binary zoneinfo files directly, if you write a directory walker, strings finder, and filter to do the same as the script. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
On Oct 15, 2017, at 1:45 PM, Michael Douglass <mikeadouglass@gmail.com> wrote:
While the tzdist protocol is http and does not (yet) handle zoneinfo format it does seem a long way towards what you're talking about.
https://tools.ietf.org/html/rfc7808
The main impediment (I believe) to handling zoneinfo data is coming up with a standard format in an rfc
I don't understand that. Standardizing encoding of interchange data is one of the two main topics for any RFC. paul
Paul.Koning@dell.com wrote:
While the tzdist protocol is http and does not (yet) handle zoneinfo format it does seem a long way towards what you're talking about.
https://tools.ietf.org/html/rfc7808
The main impediment (I believe) to handling zoneinfo data is coming up with a standard format in an rfc I don't understand that. Standardizing encoding of interchange data is one of the two main topics for any RFC.
I suspect that he was referring to the hassle of writing the RFC and getting it reviewed and so forth. Doing that has been (quite low) on my list of things to do for some time now.
Daniel Ford wrote:
This would be fine as long as my clock has access to the TZdb each year, but what if the database someday ceases to exist (or be maintained)?
Then you lose. DST rules change from year to year; even if you're only interested in current times you need frequent updates. In practice the extent of the need varies between locations, but even in the most stable cases after a few years you'd just be guessing. The database does support this kind of guessing, however. There's a nearly-POSIX-TZ field in the tzfiles, which represents the current best guess of what behaviour the zone will have after all the explicitly listed transitions. You can extract that field and process it indefinitely on the small device. -zefram
zefram, Thanks for your observations. "Then you lose." - Not quite! The user can still enter their local DST rules manually, if/when they change. "DST rules change from year to year" - In any given year, there are bound to be changes *somewhere*. But for a given fixed locality, the DST rules (in most civilised countries) tend to be pretty stable. E.g. AU/NSW, Vic, Tas have been the same since about 2008. I'm not sure why (but I don't yet understand the database) the TZdb would need to indulge in 'guessing' future DST dates/times. In most civilised jurisdictions these are legislated and published by governments, and as already mentioned, are usually pretty stable. But I'm still learning! :-) Regards, Daniel -----Original Message----- From: Zefram [mailto:zefram@fysh.org] Sent: Monday, October 16, 2017 8:59 PM To: Daniel Ford Cc: 'Time Zone Mailing List' Subject: Re: [tz] Beginner's help request Daniel Ford wrote:
This would be fine as long as my clock has access to the TZdb each year, but what if the database someday ceases to exist (or be maintained)?
Then you lose. DST rules change from year to year; even if you're only interested in current times you need frequent updates. In practice the extent of the need varies between locations, but even in the most stable cases after a few years you'd just be guessing. The database does support this kind of guessing, however. There's a nearly-POSIX-TZ field in the tzfiles, which represents the current best guess of what behaviour the zone will have after all the explicitly listed transitions. You can extract that field and process it indefinitely on the small device. -zefram
Daniel Ford wrote:
I'm not sure why (but I don't yet understand the database) the TZdb would need to indulge in 'guessing' future DST dates/times.
Future timezone rules are always guesses. `Civilised' countries may always announce changes some time ahead, but they don't actually say how far ahead this is. They don't guarantee it, and even if they did claim to give a guarantee they'd probably occasionally break it, such as with Brazil's short-notice rule change in 2002. Even if you do get a hard guarantee, it'll be for a limited period (two years?), and the rest of the future beyond that is a guess. And of course you can't ignore all the `uncivilised' places where even next week's timezone is a guess. There's also a theoretical possibility of a DST rule (firmly in place well in advance) invoking an observational version of the Islamic calendar, making a DST switch depend on actual eyeball observation of the new moon. So, overall, if you stop taking updates, then in most places you'll be OK in practice for a few years, but some places you won't. You can't tell in advance which places will be OK without an update, and there's nowhere that it's safe to go unupdated for years. -zefram
On Oct 16, 2017, at 9:25 AM, Zefram <zefram@fysh.org> wrote:
Daniel Ford wrote:
I'm not sure why (but I don't yet understand the database) the TZdb would need to indulge in 'guessing' future DST dates/times.
Future timezone rules are always guesses.
The Theory file has a lot of explanation about this. The reason is really very simple: DST rules are laws (or regulations) and like any other law, subject to change at any time at the whim of politicians. Some places make up a new rule every year, while others make up rules that are "permanent". But none of them actually are permanent, because they are always subject to revision later. When you see a "guess" that typically means the actual rule was created by the politicians as a one-year rule so we have no idea wha they will do in the future. The guess then reflects what we think they might do next year, based on what they did in the past. Sometimes the guess is wrong. As they say "past records are no guarantee of future performance". As an engineer, you might be tempted to think that DST rules have some sort of rational basis. But they don't, and that is why you need to know what the current rules are. Think of it like your tax rate: that changes at random times also. paul
Hi Daniel, One thing to keep in mind - the TZ database is prolific, and already has widespread adoption in many platforms, languages and libraries. Sure, there are many cases where one might need to go back to the sources, but most of the time there's already work of others to leverage. In the case of the NodeMCU platform, I suggest you start with the "TZ" LUA module, shown here: https://github.com/nodemcu/nodemcu-firmware/tree/master/lua_examples/timezon... You'll want to use current data files though, rather than the few that are given there. You should be able to generate them with the zic compiler from this project (though getting it working on Windows is tricky) - or you can just gather them precompiled from any tzdata distribution on a Linux system. Cheers -Matt ________________________________ From: tz-bounces@iana.org <tz-bounces@iana.org> on behalf of Daniel Ford <dfnojunk@gmail.com> Sent: Saturday, October 14, 2017 5:45 AM To: tz@iana.org Subject: [tz] Beginner's help request I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable. Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information... (I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.) I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC. The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches. For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow. Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information?? I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing. Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as... // Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld! ... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler! Any help in using the TZ database will be much appreciated (and acknowledged in my source code). Regards, Daniel --------------- Daniel Ford (Gerroa, Australia) "My loathings are simple: stupidity, oppression, crime, cruelty, soft music." - Vladimir Nabokov
On 2017-10-16 12:42, Matt Johnson wrote:
On Saturday, October 14, 2017 5:45 AM, Daniel Ford wrote:
I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable. Given that DST 'rules' can change at the whim of governments, my firmware will need to annually look up 'current' DST rules from the web. After much searching, it seems that your TZ Database is the best supported such reference for time zone rules. But I'm having great difficulty working out how I might access its mine of information... (I'll understand if you don't have time to help out a beginner, and maybe then you could refer me to a help site for TZ database applications.) I'm a hardware engineer with some programming experience, particularly assembler (various MCUs) and some C, but little or no experience in other languages. I'm developing this clock on an Arduino-like platform (NodeMCU, which has in-built Wi-Fi). My development tools run on a Win7Pro64 PC. The tools that you kindly provide for your database appear to be geared primarily towards Linux/Unix platforms, and I have no idea how I might convert them for a NodeMCU! I'm looking for some 'simple' way to access the desired TZ/DST information, preferably using simple string searches. For my application, all I'm interested in are the *current* rules. The wealth of history in your db is undoubtedly of interest to many, but for me is just 'clutter' to be searched through. But writing a program to find those rules for a particular locality has me stumped right now. I can't expect my dumb users to know a letter code for their time-zone/DST-rules, so all I'm expecting them to select from program-generated lists are: (1) their world region (Europe, Asia, Australasia, etc), and (2) the name of a major city in their time-zone with the same DST rules they follow. Let's take a concrete example: say my user has nominated 'Australasia' and 'Hobart'. So I look in your Australasia file and do a (manual at this stage) search for 'Hobart', but find only one reference to it (other than in comments). The jumble of numbers and letters there do not (clearly) tell me when DTS starts or ends. So how could I possibly write some code to find out that required information?? I've looked through the TZ_database-2017b.tar file for some sort of 'how-to' explanation of what all the file contents mean and how to programmatically search for a specific TZ, but have so far found nothing. Ideally, what I'd like to see is a text-only table of just current TZ/DST rules for each region/locality (they could all fit into a single file with all the historical data and comments removed), perhaps in the clear form used by Jack Christensen in his excellent Arduino Timezone library examples, such as... // Australia Eastern Time Zone (Sydney, Melbourne, Hobart) TimeChangeRule aEDT = {"AEDT", First, Sun, Oct, 2, 660}; TimeChangeRule aEST = {"AEST", First, Sun, Apr, 3, 600}; TimezoneExt ausET("NSW, Vic, Tas", aEDT, aEST); // Australia Eastern Time Zone (Brisbane) TimezoneExt QldT("Queensland", aEST, aEST); // no DST in Qld! ... which clearly tells me in a program-searchable way that in the main eastern states DST starts on the first Sunday in October at 2am, becoming UTC offset +11 hours (660 minutes), and ends on the first Sunday in April at 3am, reverting to UTC offset +10 hours. Couldn't be simpler! Any help in using the TZ database will be much appreciated (and acknowledged in my source code).
One thing to keep in mind - the TZ database is prolific, and already has widespread adoption in many platforms, languages and libraries. Sure, there are many cases where one might need to go back to the sources, but most of the time there's already work of others to leverage.> In the case of the NodeMCU platform, I suggest you start with the "TZ" LUA module, shown here:> https://github.com/nodemcu/nodemcu-firmware/tree/master/lua_examples/timezon... You'll want to use current data files though, rather than the few that are given there. You should be able to generate them with the zic compiler from this project (though getting it working on Windows is tricky) - or you can just gather them precompiled from any tzdata distribution on a Linux system. Also available by installing Cygwin and/or Ubuntu for Windows, which include tzdata in their base installs, and using Windows File Explorer to search for the zoneinfo directory.
-- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Zefram/Paul, Thanks for your further comments. I accept that DST 'rules' can be ephemeral, particularly in some locations. But I think the days of 'making a DST switch depend on actual eyeball observation of the new moon' are perhaps past, at least in most modern Muslim nations. These days they tend to accept the science of celestial motion, which predicts moon (or whatever) appearances at specified locations with great accuracy. You only have to Google 'ramadan 2018' (without the quotes) to convince yourself of this. If you know of other less predictable events that routinely cause changes to DST start/end dates, I'd be interested to hear about them. Nevertheless, I understand that DST rules *do* change (and even timezones can change, though more rarely), so my intention is certainly to try to access the TZdb (or a derivative thereof) for my clock. But I would still prefer to download the current 'rule' rather than the current dates, so I can continue calculating the changeover events in the (unlikely) future absence of the TZdb. In that unlikely event, my clock would continue using the most recent rule until the user's local authority decided to change the rule, at which time the user would have to manually enter the new rule into the clock. Hopefully this won't happen within a reasonable product lifetime! I still have a lot of reading to do about the TZdb (thanks for the 'Theory' pointer, Paul), but when I've done that I'll almost certainly be back with a lot more questions. Than you all for your patience and tolerance. Regards, Daniel
Matt, Thanks for the TZ.lua Github link. Though I found that page was not presented in the format I'm used to on Github. ALL the files on that page appear to be in HTML format rather than what they purport to be, making it hard to use. I managed to convert the .lua and .md files to real code and formatted text (resp.), but the .zone files elude me. Not that I'd use them (as you suggested I don't), but I just wanted to view their structure. But even renaming one as .html, it did crazy things in both Firefox and IE, and would not display the contents in any form I could recognise. I also tried downloading the 'raw' version, but it was not ASCII text. I am aware that there are 'derivative' versions of the TZdb. But after many decades of experience, I have little faith in the longevity of any specific website. And I would have less faith in a site established/maintained by just one person/tiny company than I would in one maintained/sponsored by a large organisation (like the TZdb is). Remember I'm talking about a hardware product, which I would hope might have a 20-30 year lifetime (though with the low-cost Chinese-manufactured NodeMCU, that might be wishful thinking!), and I don't want to have to be sending firmware updates to users whenever a requisite website disappears. Regards, Daniel
On Oct 16, 2017, at 9:58 PM, Daniel Ford <dfnojunk@gmail.com> wrote:
... Nevertheless, I understand that DST rules *do* change (and even timezones can change, though more rarely), so my intention is certainly to try to access the TZdb (or a derivative thereof) for my clock. But I would still prefer to download the current 'rule' rather than the current dates, so I can continue calculating the changeover events in the (unlikely) future absence of the TZdb.
I'm not sure why you got the impression that the TZDB gives you transition dates, rather than the rules you were very sensibly looking for. The TZDB is exactly that: a set of rules. The latest one for any location is usually of the form "DST ends on the first sunday of november at 2:00 am local time, until further notice". All historic rules (no longer current) of course have a start and end date. So an application that uses the TZDB always does what you describe: it uses the rule it has been given to calculate all future times. The reason new TZDB releases occur perhaps a dozen times per year is precisely that the old rule has been superseded by a new political act. If you aren't up to date, you'll be in trouble anywhere from months to many years out, depending on your location -- because some change annually (occasionally even more often) while others remain stable for years or even decades. paul
On 17 October 2017 at 10:23, <Paul.Koning@dell.com> wrote:
The TZDB is exactly that: a set of rules.
Precisely. If you're more interested in the rules than individual dates, you probably want to consume the data files directly (africa, antarctica, asia, etc.) and then use the rules therewithin to compute dates as far out as you need. (zic is the tool developed for this purpose.) Just keep in mind, for all the reasons already mentioned, that you should check regularly for updates whenever possible. On 16 October 2017 at 08:32, Daniel Ford <dfnojunk@gmail.com> wrote:
The user can still enter their local DST rules manually, if/when they change.
There are a handful of zones for which this can be more complicated than initially meets the eye. The cases surrounding the dates of observance of Ramadan are certainly one such case, though as you note, most such areas have some sort of *de facto* "buffer zone" around the month for DST purposes, so the actual lunar observation is less important. But also note that this means many of these locations are actually changing their clocks (possibly) four times a year, and according to two different calendar systems. This is something that tz manages to approximate well, but that would be difficult to expect a typical end-user to program into an embedded device, if the underlying rules were to be tweaked. -- Tim Parenti
On 2017-10-16 19:58, Daniel Ford wrote:
I accept that DST 'rules' can be ephemeral, particularly in some locations. But I think the days of 'making a DST switch depend on actual eyeball observation of the new moon' are perhaps past, at least in most modern Muslim nations. These days they tend to accept the science of celestial motion, which predicts moon (or whatever) appearances at specified locations with great accuracy. You only have to Google 'ramadan 2018' (without the quotes) to convince yourself of this. If you know of other less predictable events that routinely cause changes to DST start/end dates, I'd be interested to hear about them.
Nevertheless, I understand that DST rules *do* change (and even timezones can change, though more rarely), so my intention is certainly to try to access the TZdb (or a derivative thereof) for my clock. But I would still prefer to download the current 'rule' rather than the current dates, so I can continue calculating the changeover events in the (unlikely) future absence of the TZdb.
In that unlikely event, my clock would continue using the most recent rule until the user's local authority decided to change the rule, at which time the user would have to manually enter the new rule into the clock. Hopefully this won't happen within a reasonable product lifetime!
I still have a lot of reading to do about the TZdb (thanks for the 'Theory' pointer, Paul), but when I've done that I'll almost certainly be back with a lot more questions. Than you all for your patience and tolerance.
Just look at the release history of a dozen or two releases per year, because governments changed time zone and daylight saving time with little notice. You should have been aware of Turkey making a change with zero notice last year. Some small countries, especially Caribbean nations, seem especially prone to give short notice annually. Someone from Sudan just posted they are changing their GMT offset at the start of next month. And people complain their mobiles and PCs don't show the correct time, when list members have to notice articles in the press, search for something like official confirmation, then the rule changes have to be made and posted on here, picked up by the mobile OS or PC distribution vendors, packaged, and made available for update; those updates then have to be picked up and installed by individuals or support staff; and in many cases pushed out across systems, or by mobile networks, to get to the end user's systems and applications. I suspect the long term solution may have to require Internet treaties with ICANN country members, and contracts with country code domain registrars, requiring their governments provide official advance notice of time zone and daylight saving time changes with at least 6 months notice to IANA, similar to their requirements to do so to IATA, to avoid air route, airport, facilities, and staff scheduling issues. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Paul & Tim, thank you for your examples and explanations. I'm finally starting to get an understanding of the scope of the complex problem. I've also read some of the 'explanatory' files to better understand the database contents, POSIX format, etc, and I think I'm closer to understanding how to use the database for my clock. It's really interesting that some countries/localities still make short-term changes to DST start/stop dates, and that some areas experience four changes in a year! My first reaction was: "I won't offer my clock to those countries", but then I thought: "But they're the ones who need it even more than me!" So I need to try to deal with those weird rule changes in my clock. My first thought is: "How does the TZ database record rules when they change four times in a year rather than two?" My next problem was "where do I find these countries in the database, since there's no 'MiddleEast' file?" But then I was alerted to Zone1970.tab, and found that they were included in the 'Asia' file. So I went looking for a place where these quadruple rules might apply. My first search was for Baghdad, which turned up another issue in the current (2017b) Asia file that doesn't quite make sense to me. From line 1146... Rule Iraq 1986 1990 - Mar lastSun 1:00s 1:00 D # IATA SSIM (1991/1996) says Apr 1 12:01am UTC; guess the ':01' is a typo. # Shanks & Pottenger say Iraq did not observe DST 1992/1997; ignore this. # Rule Iraq 1991 2007 - Apr 1 3:00s 1:00 D Rule Iraq 1991 2007 - Oct 1 3:00s 0 S # Zone NAME GMTOFF RULES FORMAT [UNTIL] Zone Asia/Baghdad 2:57:40 - LMT 1890 2:57:36 - BMT 1918 # Baghdad Mean Time? 3:00 - +03 1982 May 3:00 Iraq +03/+04 Because there is no *current* RULE for Baghdad (the last expired 2007 according to the table), I assume that DST no longer occurs there. But Bill Seymour's explanation of the digits in the FORMAT column are 'standard time name/daylight saving time name'. Given that DST no longer applies (if my Rule interpretation is correct) and the standard GMT offset is +3h, why would these (now identical?) zones be named +03 and +04? At this stage I haven't continued on my search for a quad-change locality (I'll wait till I better understand the 'simple' ones, such as the above, first!), but if anyone can direct me to a quad-change rule, it would save me some time further down the track. Regards, Daniel
On Oct 18, 2017, at 8:58 AM, Daniel Ford <dfnojunk@gmail.com> wrote:
... So I went looking for a place where these quadruple rules might apply. My first search was for Baghdad, which turned up another issue in the current (2017b) Asia file that doesn't quite make sense to me. From line 1146...
Rule Iraq 1986 1990 - Mar lastSun 1:00s 1:00 D # IATA SSIM (1991/1996) says Apr 1 12:01am UTC; guess the ':01' is a typo. # Shanks & Pottenger say Iraq did not observe DST 1992/1997; ignore this. # Rule Iraq 1991 2007 - Apr 1 3:00s 1:00 D Rule Iraq 1991 2007 - Oct 1 3:00s 0 S # Zone NAME GMTOFF RULES FORMAT [UNTIL] Zone Asia/Baghdad 2:57:40 - LMT 1890 2:57:36 - BMT 1918 # Baghdad Mean Time? 3:00 - +03 1982 May 3:00 Iraq +03/+04
Because there is no *current* RULE for Baghdad (the last expired 2007 according to the table), I assume that DST no longer occurs there.
I would not call that "no current rule". The lines in the rule files specify offsets, and the time when that offset goes into effect. For any given time, the offset that applies is the one specified by the most recent prior effective-time. Your conclusion looks correct to me: since the most recent time change established standard time (on 10/1/2007) has not applied since then. paul
Hi Brian, At 12:24 PM 17-10-2017, Brian Inglis wrote:
I suspect the long term solution may have to require Internet treaties with ICANN country members, and contracts with country code domain registrars, requiring their governments provide official advance notice of time zone and daylight saving time changes with at least 6 months notice to IANA, similar to their requirements to do so to IATA, to avoid air route, airport, facilities, and staff scheduling issues.
Why was ICANN country members or country code domain registrars chosen in the above? Regards, -sm
On 2017-10-18 08:07, SM wrote:
At 12:24 PM 17-10-2017, Brian Inglis wrote:
I suspect the long term solution may have to require Internet treaties with ICANN country members, and contracts with country code domain registrars, requiring their governments provide official advance notice of time zone and daylight saving time changes with at least 6 months notice to IANA, similar to their requirements to do so to IATA, to avoid air route, airport, facilities, and staff scheduling issues.
Why was ICANN country members or country code domain registrars chosen in the above?
Relationship between countries and CC domains on one hand, and ICANN and IANA on the other, similar to national airlines and IATA on the gripping hand. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
On 10/18/2017 07:07 AM, SM wrote:
Why was ICANN country members or country code domain registrars chosen in the above?
Anything of this scope would be like the WIPO Internet Treaties, and would be a big deal: in the US it would require ratification by Congress. It would bind any country signing such a treaty, and presumably this would eventually supersede ICANN. The most recent attempt to formulate such an Internet treaty featured a controversial proposal from a group that included China, Russia, Sudan, and others. These countries wanted to manage the IANA, and presumably this would have included managing the tzdb project. It would have meant for a lot more politics and bureaucracy for tzdb. Had this happened, I probably would have moved the tzdb project to a safer home outside IANA. Luckily, saner heads prevailed and the draft treaty went down in flames. Where treaties are concerned, be careful what you wish for. For more, see: Garber M. How the UN's 'game-changing' Internet treaty failed. The Atlantic 2012-12-14. https://www.theatlantic.com/technology/archive/2012/12/how-the-uns-game-chan...
The only "current" example of this is Africa/Casablanca (Morocco). There's a good visualization here: https://www.timeanddate.com/time/zone/morocco/casablanca -Matt ________________________________ At this stage I haven't continued on my search for a quad-change locality (I'll wait till I better understand the 'simple' ones, such as the above, first!), but if anyone can direct me to a quad-change rule, it would save me some time further down the track.
Daniel Ford wrote:
Because there is no *current* RULE for Baghdad (the last expired 2007 according to the table), I assume that DST no longer occurs there.
Essentially, yes. The last transition in the listed rules is the transition from DST to standard time in October 2007, and since then Baghdad has been on UT+3h. Whether this amounts to having "no current rule" is debatable, though: it depends on whether you regard the extent of the rule as being the instantaneous transition or the period of time between transitions. In the latter interpretation, the current rule is the one that says save=0 from October 2007. The current part of the Asia/Baghdad zone definition still says that the "Iraq" ruleset applies.
Given that DST no longer applies (if my Rule interpretation is correct) and the standard GMT offset is +3h, why would these (now identical?) zones be named +03 and +04?
This is not two zones, it is and always was one zone. With DST no longer happening, the "+03/+04" has the effect that the zone's short name is always "+03". The "+04" part of the name format just doesn't get used from 2008 onwards. It still needs to be specified, though, because that zone definition line applies from 1982 onwards, and so also covers times that (per the "Iraq" ruleset) did have DST.
At this stage I haven't continued on my search for a quad-change locality
Africa/Casablanca. -zefram
Thanks guys - you've all been extremely helpful, and I appreciate it. I've been trying to conceive an algorithm for finding the 'current' rules for a locale, based on its 'reference city', and you've helped clarify what that will involve, though I'm not there yet. Thanks also for the pointer to Africa/Casablanca as one of the known '4-transitions' localities. And I liked the timeanddate visualisation, Matt. (Timeanddate has long been my 'goto' site for world time info and calculators.) So now I've opened the 'africa' file and searched for Africa/Casablanca, but I only see *two* rules for 2017. So how does the TZdb indicate that there are *four* transitions this year? Regards, Daniel
Daniel Ford wrote:
I've opened the 'africa' file and searched for Africa/Casablanca, but I only see *two* rules for 2017.
There are four rules: Rule Morocco 2013 max - Oct lastSun 3:00 0 - Rule Morocco 2014 2021 - Mar lastSun 2:00 1:00 S ... Rule Morocco 2017 only - May 21 3:00 0 - Rule Morocco 2017 only - Jul 2 2:00 1:00 S
On Wed, Oct 18, 2017, at 20:08, Daniel Ford wrote:
So now I've opened the 'africa' file and searched for Africa/Casablanca, but I only see *two* rules for 2017. So how does the TZdb indicate that there are *four* transitions this year?
There are also 2014-2022 and 2013-2035 rules, for the regular DST transitions in March and October respectively. (The other transitions represent a 'hiatus' in DST for Ramadan, which can't be represented easily in TZdb format, therefore why there are lines for every year.)
On 2017-10-18 18:08, Daniel Ford wrote:
Thanks guys - you've all been extremely helpful, and I appreciate it. I've been trying to conceive an algorithm for finding the 'current' rules for a locale, based on its 'reference city', and you've helped clarify what that will involve, though I'm not there yet.
That work is already done in zic where stringzone() generates a POSIX envvar. You could cut down zic to do only what it needs to generate a POSIX envvar in stringzone, and output the zone name and envvar as your desired end product. You are only interested in the zones in zone1970.tab and tzselect shell script shows what you need to do for zone selection. The UI is the only place you may need zone names, but RC and GPS world time zone watches just use 48 airport codes for selection and display. Handling a POSIX TZ string is shown in Bill Plauger's "The Standard C Library", but will need updated to support new POSIX <...> quoting syntax used in tz.
Thanks also for the pointer to Africa/Casablanca as one of the known '4-transitions' localities. And I liked the timeanddate visualisation, Matt. (Timeanddate has long been my 'goto' site for world time info and calculators.)
So now I've opened the 'africa' file and searched for Africa/Casablanca, but I only see *two* rules for 2017. So how does the TZdb indicate that there are *four* transitions this year?
-- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Paul, Ah, thanks for clearing up my oversight - I must be getting myopic in my old age! ;-) My eyes only moved up as far as the '2017' entries and didn't go further. Now I can perhaps complete my algorithm, which I might post later for comment. Regards, Daniel
G'day guys. Sorry for the long absence - had a quest staying over a week, which interrupted my hobby project! After lots of reading of various mentioned info files, I think I have a little understanding of the structure of the TZdb now, and how I might make use of it to update the DST rules in my proposed clock. Would you please look over the description below and tell me if I've got it right, or otherwise! ----- Possible method... User selects: First: region, from presented list. Then program displays list of cities within that region, and user picks one. We then have a region/city string, e.g. 'Africa/Casablanca'. Then we need to download the current 'africa' file (for the above example) from the TZ FTP page, and save it in SPIFFS. We then do a string search in that file, for (in this example) "Zone Africa/Casablanca" (where the space between Zone and Africa can be *any* whitespace, though typically a tab character). Move forward through the file, line by line, till you find the last line of that zone entry (it's UNTIL column is empty, [and maybe it's always followed by an empty line?]). Note the GMTOFF and name of the rule. Now move backwards through the file, line by line, till you find a Rule line with that name, having: a FROM date* <= the current date, AND a TO date* >= the current date, OR 'max', OR '-' or 'only' iff FROM == current date (* 'date' will often be just a year, in which case the comparison is with the current year) For each such line, store the IN, ON, AT and SAVE values. When sequenced in date order, these become the current DST rules for this reference city. If no Rule line meets these criteria, take GMTOFF to be the offset throughout the year. ----- Also, if anyone's interested, I have posted a simple demo of some of the hardware that will be used in my clock (the NodeMCU in particular). I won't be using the display in that video (way too small), and of course I need to add the TZdb stuff to my code, so DST rules are continually updated, rather than hard-coded as in that demo. And my proposed clock will only display accurate local time, not multiple zones. https://youtu.be/PyDerVMuXDI Regards, Daniel --------------- Daniel Ford (Gerroa, Australia)
On 10/30/2017 05:30 PM, Daniel Ford wrote:
Then we need to download the current 'africa' file (for the above example) from the TZ FTP page, and save it in SPIFFS.
We're not set up to support lots of users downloading files. The intent is that you download the information once, and then resend it to your users.
We then do a string search in that file, for (in this example) "Zone Africa/Casablanca" This sounds pretty brittle. There should be a better way; unfortunately we don't support that "better way" right now. But it's something that you could do, since you can massage the data before sending it to your users.
Daniel Ford wrote:
User selects: First: region, from presented list. Then program displays list of cities within that region, and user picks one.
You didn't say where to get the lists of regions and cities. The list of regions is pretty fixed, but the list of cities routinely changes, mainly by adding new cities when localities that previously had identical zone behaviour (since 1970) diverge. You need to acquire the list of zones in a dynamic manner.
Then we need to download the current 'africa' file
We don't guarantee anything about the names and content of specific data source files. You can extract the list of data source files from the Makefile, by a bit of trickery: $ make check_links AWK='@echo x' | sed -n '1s/^x -f checklinks\.awk //p' africa antarctica asia australasia europe northamerica southamerica etcetera backward pacificnew systemv factory or you can use the generated source-format data file tzdata.zi, which contains everything. But parsing the source-format data files yourself is relatively hairy. You should prefer to use zic to generate the compiled data files, and extract what you need from them. The binary format of the compiled files is simpler, stabler, and provides the information you want more directly.
from the TZ FTP page
It would be a bad idea to have mass-produced consumer devices all hitting the canonical FTP or HTTP sites. You should arrange distribution to these devices yourself, at minimum by having them query a mirror that you operate.
Move forward through the file, line by line, till you find the last line of that zone entry (it's UNTIL column is empty, ... these become the current DST rules for this reference city.
No. The last line of the Zone entry provides the latest known rules for the zone, but those are not necessarily the current rules. They may be future rules that have been announced but not yet come into effect. If you want to know the rules for any specific time, including the present and near future, you *must* check the UNTIL column to find out which line of the Zone entry is applicable. But more generally, parsing within a source-format data file to find the specific part of a zone's data that you want sounds like a bad idea. Even given that you're parsing the source format, this is an especially tricky way to go about it. Better to compute the whole zone's behaviour and extract the bit that you want at a later stage of processing.
If no Rule line meets these criteria, take GMTOFF to be the offset throughout the year.
No. If no Rule line is `current' by that definition, then the SAVE value of the most recent pre-`current' rule applies. Indeed, even if there are `current' Rule lines, if the time of the first one has not yet arrived then the SAVE value comes from an older rule, not from any of the `current' ones. It is not correct to divide rules into `current' and non-`current' in the way that you describe. The way to understand Rule lines is that they describe a series of transitions and (part of) the state of the zone between those transitions. There is no artificial distinction between one year's state and transitions and another year's; there's just a sequence of transitions, which can occur at any time. Where a Rule line specifies a range of years, then that's a formula for constructing multiple transitions. Transitions are instantaneous events, and so are never meaningfully "current". The only thing that's current is the state of the zone, which is whichever state applies between the most recent transition in the past and the nearest transition in the future. That state will be specified by the Rule line that defines that most recent past transition. For your clock, you're probably best off not attempting to have the clock store rules that can be applied indefinitely far in the future. Better to compute a few years of explicit future transitions, and have the clock just work its way through the list, eventually erroring if it's gone years without updating its knowledge. -zefram
Paul, Thanks for your comments, which come as a bit of a shock to me! Given that the TZdb is possibly the only *source* data on time zones/DST that is updated periodically (most other 'sources' are *derived from* the TZdb), I would have expected that lots of people would wish to access it when they have a need for current DST information (such as all those derivative websites). "The intent is that you download the information once, and then resend it to your users." But this contradicts the statement in README that says: "It is updated periodically to reflect changes made by political bodies to time zone boundaries, UTC offsets, and daylight-saving rules." That infers that users (including all those derivative websites) must *periodically* access the TZdb data to ensure the rules they use are current! Especially since, as David Patte pointed out, some jurisdictions change their rules multiple times in a year, with little notice! "...since you can massage the data before sending it to your users." I am not set up to send anything, other than posting a piece of hardware (my clock) to a purchaser. After that, it's up to the clock to keep itself updated on TZ/DST rules for its destination locale, using its wi-fi access to a suitable information source (as well as regularly checking UTC time from an NTP server, of course). It was my intention that the clock would access the TZdb about once per week to check the latest rules. And all the clocks (assuming I sell more than a few!) will access at different times. The intention is to program the clock to check the TZdb every, say, 169 hours since boot-up (the extra hour ensures it doesn't happen at the same time every week, as further 'randomisation'). Another reason I wouldn't want to set up a website to serve these clocks is that this is just a hobby project (though I hope to sell a few clocks via eBay), and any website I set up would only last as long as I do. While I don't expect to die in the next few years (I'm only 73!), it could happen, and where would that leave those clock owners? More about this in my response to Zefram... Regards, Daniel
Daniel Ford wrote:
"The intent is that you download the information once, and then resend it to your users." But this contradicts the statement in README that says: "It is updated periodically to reflect changes made by political bodies to time zone boundaries, UTC offsets, and daylight-saving rules." That infers that users (including all those derivative websites) must *periodically* access the TZdb data
This does not mean that all users must access the iana.org servers directly. iana.org is not set up to handle that kind of load! All it means is that users should get the data somehow. In practice, Red Hat users get copies of the data from Red Hat, Apple users get copies from Apple, and so forth. IANA cannot support billions of devices all downloading data directly from IANA.
"...since you can massage the data before sending it to your users." I am not set up to send anything, other than posting a piece of hardware (my clock) to a purchaser.
Then I'm afraid you'll need to change your setup. Sorry, but IANA has limited resources and it cannot be the direct central server for all the clocks on the planet. It should be easy for you or your successors to maintain a server for your customers' clocks, a server that is a clone of the IANA server.
Daniel, as suggested by Paul, I do in fact download the information once, then provide my copy to my own users. On 2017-10-31 21:47, Daniel Ford wrote:
Especially since, as David Patte pointed out, some jurisdictions change their rules multiple times in a year, with little notice!
--
Zefram, "You didn't say where to get the lists of regions and cities." I had planned to hard-code these from zone1970.tab into the clock, but since you've mentioned that they sometimes change too, I'll now have to download that file (whenever it changes) to the clock too. I've never coded FTP into an embedded system before, but I'm hoping there will be some way the FTP routines (in the Arduino FTP library I've downloaded) can determine (without downloading) whether a file has changed since 'the last time' (e.g. by date-stamp). "We don't guarantee anything about the names and content of specific data source files." I had naively assumed that there was a 1:1 correspondence between region names in zone1970.tab and file names containing zones/rules for those regions. Is this not correct? If not, I would have to (impertinently) ask: "Why not?" :-) I think I mentioned in one of my early mails that I am a PC/Windoze user, so unfortunately all the code tools you provide for the TZdb are of little use to me. That's why I'm trying to tackle the *source* (ASCII) data directly, rather than any binary files. It might be hard, but if all the required information resides in those text files, it must be possible to tease it out, somehow. That was my aim with my suggested 'algorithm'. "It would be a bad idea to have mass-produced consumer devices all hitting the canonical FTP or HTTP sites." OK, I'm getting the impression that the TZdb site is bandwidth-limited. But I'd be very surprised (and pleased!) if I sold more than 100 clocks, and surely the TZdb site would handle 100 FTP requests spread randomly over a week? In the event that my clock became a runaway success and sold many more, I would then look at setting up a site to serve data to them. But that becomes difficult, since I'm opposed in principle to any 'rental' schemes for hardware (and software for that matter, though I reluctantly live with that reality), so would have to pay for the running of that website from one-off profits - not a good recipe in the long-term. Anyway, I think that's an unlikely scenario. "...you *must* check the UNTIL column to find out which line of the Zone entry is applicable." Understood - thanks for the pointer. So that part of my algorithm should change to say: ----- Move forward through the file, line by line, till you find a line of that zone entry whose UNTIL column contains a post-current date, or (if none with such a value) is empty. ----- "But more generally, parsing within a source-format data file to find the specific part of a zone's data that you want sounds like a bad idea." I agree - it's a horrendous job, made difficult by the particular way the database data is presented, with no file just containing 'current' rules in a simple format. This is obviously an historic inheritance, and the database has presumably grown 'like topsy' to be the way it is today. Of course the clever people who maintain it have come up with complex code to process and extract the data in a simpler form, but that is not available to me. I had (perhaps forlornly) hoped that someone could set up some code (run whenever the database contents changed) to do that extraction and create a single 'current' file that lists all zones and DST rules *at the current time*, and include that results file in the database for access by non-Unix folk like me. But if that can't happen, I'm stuck with parsing within source-format data files. "If no Rule line is `current' by that definition, then the SAVE value of the most recent pre-`current' rule applies." Let me see if I understand this... Are you saying that some jurisdictions will make a *permanent* change to DST (+1hr from 'standard' time, for example) *rather than* officially changing their GMT-offset? "Transitions are instantaneous events, and so are never meaningfully 'current'." I understand that. What I mean by a 'current' rule is: "this rule applies to the DST transitions for this year." For me, a 'current' rule does not define the current time; it defines the expected near-future *changes* in GMT-offset (to/from DST). This is all I need - the 'current' DST rules. The clock's firmware then monitors those rules and makes the offset changes when the corresponding UTC occurs, as you saw in the video I linked earlier. "Better to compute a few years ..., eventually erroring if it's gone years without updating its knowledge." But what then? It will then need to access the TZdb to update its rules, so we're back to where I started - the need to check occasionally for updated rules. I agree that in most 'stable' jurisdictions there will be no changes for years, and in this case the clock would just check if the rules for its installation locale have changed (by date-stamp?), and only download the new files if they have. But to allow for clocks being bought and installed in 'unstable' (time-rule) jurisdictions, I would want the clock to check for updates every week or so. Paul, "Then I'm afraid you'll need to change your setup. Sorry, but IANA has limited resources and it cannot be the direct central server for all the clocks on the planet. It should be easy for you or your successors to maintain a server for your customers' clocks, a server that is a clone of the IANA server." Understood. As mentioned, this is not a 'commercial' project in the normal sense. It's a hobby project (for my own use) that I hope might make a little pocket money from eBay sales too. It's unlikely my children would be interested in maintaining my server. And the hard part, for me, would be setting up the server in the first place, as I have negligible experience in that department. I'll give my approach more thought after I've mastered the interpretation of the TZdb and how to extract the desired info from it. Regards, Daniel
Daniel Ford wrote:
I'm hoping there will be some way the FTP routines (in the Arduino FTP library I've downloaded) can determine (without downloading) whether a file has changed since 'the last time' (e.g. by date-stamp).
There are optional parts of the FTP protocol that can give timestamps, but timestamps are relatively likely to get clobbered by accident. A better method of checking for an update of tzdb is to look at the version number.
I had naively assumed that there was a 1:1 correspondence between region names in zone1970.tab and file names containing zones/rules for those regions.
There is not. There's an approximate correspondence for most regions, but notably there's no "pacific" file, with most (but not all) Pacific/ zones being in the "australasia" file.
"Why not?" :-)
Convenience of maintenance sometimes dictates exceptions, and in those cases political affiliations have some influence, mainly through the medium of shared DST rules. Russia spans both Europe and Asia, each of its zones is assigned to one region or the other, but they share DST rules, so for maintainability we have them all in one file, arbitrarily the "europe" file. Pacific/Honolulu, for Hawaii, is in the "northamerica" file along with the rest of the US, though since it's never observed DST that might be more a geographical decision. There's also an inertial effect, that we prefer not to change historical arrangements that don't adversely affect the output. The assignment of zones to source files is not an advertised product of the database; it's an internal arrangement.
I'd be very surprised (and pleased!) if I sold more than 100 clocks,
A hundred wouldn't be a worry, but by the time you've sold more it's too late to reconfigure the ones you've already sold. You need to at least have a mechanism for retargeting the clocks, built into all of them from the start. You also need to think about failure modes of your control mechanism. Learn from the mistakes that have been made in the past. If you wanted to do hardware engineering, rather than network engineering, you picked the wrong project.
that becomes difficult, since I'm opposed in principle to any 'rental' schemes for hardware
It's not ideal, but at least, as the person selling the clocks, you have an income that rises in proportion to the demand from the clocks. You are the person best placed to provision a mirror with commensurate capacity.
And the hard part, for me, would be setting up the server in the first place, as I have negligible experience in that department.
Happily, you can hire people to do that bit. Isn't this "money" stuff amazing. It would obviously be better if there were some distribution mechanism available that didn't depend on a single FTP or HTTP server. There's an IETF working group looking at timezone distribution issues; you should have a look to see whether they're addressing your use case. In addition to the bandwidth issues, maybe they'll handle distributing binary tzfiles, so that you don't have to parse the source.
I agree - it's a horrendous job, made difficult by the particular way the database data is presented, with no file just containing 'current' rules in a simple format. This is obviously an historic inheritance,
No, even if we designed the source format anew today it would be arranged something like this. These are *source* files, which are, and should be, arranged for convenience of maintenance. This is why we have a separate format, the binary tzfile, which is designed for convenience in looking up the offset that's in effect at a specified time. Compilation isn't just the way things are, it's a good idea.
Let me see if I understand this... Are you saying that some jurisdictions will make a *permanent* change to DST (+1hr from 'standard' time, for example) *rather than* officially changing their GMT-offset?
It has been known. For about four years we had America/Argentina/San_Luis described as year-round DST, though we've now redescribed that as a change of base offset with no DST. It also often happens, especially where DST transitions are selected from year to year, that it's not clear whether a change of offset is DST or a change of base offset, so we have to guess. Also, sometimes a southern-hemisphere DST rule says that DST ends in a particular year and never restarts. This doesn't involve year-round DST, just a regular decision to abandon DST. If you only look at that year's transition and the post-transition state, you won't know about the DST that is in effect at the beginning of the year. Your assumption that the DST status of the zone will be the same at the beginning and end of each year is unwarranted. DST is often cyclical, but the exceptions are far too numerous to count on it.
But what then? It will then need to access the TZdb to update its rules,
Yes, of course it should update. Erroring is what would happen *if it fails to update* for years on end. -zefram
On Sat, Oct 14, 2017 at 8:45 AM, Daniel Ford <dfnojunk@gmail.com> wrote:
I'm looking to develop a 'world clock' that could be sold to anyone (with Internet access via a Wi-Fi AP) anywhere in the world, that will display their local time, taking into account DST if/when applicable.
G'day Daniel, There are literally billions of computers, smart phones, and other devices in the world that do exactly this. So, a high-level question: Why do you believe that your clock requires new mechanisms? Bradley
Bradley, "Why do you believe that your clock requires new mechanisms?" That's a valid question! I'm not sure that I'm looking for a 'new mechanism', but rather a simple way of doing the same thing (on the NodeMCU platform) those billions of devices do, without any 'incremental' ongoing costs. By that I mean no additional costs over the data plan the user currently has for their wi-fi Internet connection. Your next question might then be: "If they have an Internet connection, why can't they just look at their PC/smartphone/whatever for the current time?" I'm going to tell you why *I* want such a clock, and I'm assuming I'm not the only person in the world in this situation (if I am, then it's going to be a very short production run! ;-) I'm sitting in my living room watching TV. A show that I want to watch more than the one I'm currently watching starts on another channel at 9pm. I need to know when the time approaches 9pm. My PC is downstairs in my home-office, so I'm not running down there to find the time. I might or might-not have a smartphone in my pocket, but even if I did, what a drag to have to pull it out and press a button just to see the current time. Much easier to just glance across at the table/wall clock to see the current time. This is a very trivial example, but illustrates the type of problem I'm trying to solve. There would be lots of people (particularly elderly people) without a smartphone, but with an Internet connected PC, though not in their living room/bedroom/kitchen/wherever they want to know the time. Regards, Daniel
On Thu, Nov 2, 2017 at 2:17 AM, Daniel Ford <dfnojunk@gmail.com> wrote:
I'm not sure that I'm looking for a 'new mechanism', but rather a simple way of doing the same thing (on the NodeMCU platform) those billions of devices do, without any 'incremental' ongoing costs. By that I mean no additional costs over the data plan the user currently has for their wi-fi Internet connection.
It seems, then, that your device could simply pull (compiled) zoneinfo files (from some source you've setup or arranged to use). Then you can use any zoneinfo-aware "localtime()" implementation to convert the absolute time to the local civil time.
I'm sitting in my living room watching TV. A show that I want to watch more than the one I'm currently watching starts on another channel at 9pm.
Get a PVR and watch both!
This is a very trivial example, but illustrates the type of problem I'm trying to solve. There would be lots of people (particularly elderly people) without a smartphone, but with an Internet connected PC, though not in their living room/bedroom/kitchen/wherever they want to know the time.
An even simpler design then would be to install software on the PC that would deliver the local time to your clock over the LAN. Get out of the TZ business altogether.
Bradley, "Then you can use any zoneinfo-aware 'localtime()' implementation to convert the absolute time to the local civil time." I've not yet found an Arduino library with such a function. They all seem to operate from text data. But I'll keep looking. "Get a PVR and watch both!" I have a PVR that can record up to four programs simultaneously, but sometimes even that is not enough! Seriously though, the TV example was, as I said, trivial. There are many scenarios where one would want to know the exact time and not be near a PC or smartphone. "An even simpler design then would be to install software on the PC that would deliver the local time to your clock over the LAN." That's quite possible for me alone, but not for a commercial clock. The purchaser might have any of a number of different 'computer' products in their home, including Android tablets and machines from crApple. I am certainly not about to write software for the dozens of variants that might be found in individual homes! That would also require that their 'computer' be on 24/7, assuming the user wants to know the accurate time at any time of the day or night. Regards, Daniel ________________________________ From: c410bw02@gmail.com [mailto:c410bw02@gmail.com] On Behalf Of Bradley White Sent: Thursday, November 02, 2017 6:02 PM To: Daniel Ford Cc: Time zone mailing list Subject: Re: [tz] Beginner's help request On Thu, Nov 2, 2017 at 2:17 AM, Daniel Ford <dfnojunk@gmail.com> wrote: I'm not sure that I'm looking for a 'new mechanism', but rather a simple way of doing the same thing (on the NodeMCU platform) those billions of devices do, without any 'incremental' ongoing costs. By that I mean no additional costs over the data plan the user currently has for their wi-fi Internet connection. It seems, then, that your device could simply pull (compiled) zoneinfo files (from some source you've setup or arranged to use). Then you can use any zoneinfo-aware "localtime()" implementation to convert the absolute time to the local civil time. I'm sitting in my living room watching TV. A show that I want to watch more than the one I'm currently watching starts on another channel at 9pm. Get a PVR and watch both! This is a very trivial example, but illustrates the type of problem I'm trying to solve. There would be lots of people (particularly elderly people) without a smartphone, but with an Internet connected PC, though not in their living room/bedroom/kitchen/wherever they want to know the time. An even simpler design then would be to install software on the PC that would deliver the local time to your clock over the LAN. Get out of the TZ business altogether.
On Thu, Nov 2, 2017 at 3:31 AM, Daniel Ford <dfnojunk@gmail.com> wrote:
"Then you can use any zoneinfo-aware 'localtime()' implementation to convert the absolute time to the local civil time."
I've not yet found an Arduino library with such a function. They all seem to operate from text data. But I'll keep looking.
The library discussed in this list includes source code for one.
There are many scenarios where one would want to know the exact time and not be near a PC or smartphone.
One may not be near another clock either. But, I'm not here to debate the market for your device ... I'm just trying to describe ways you might use the TZ database as it exists.
That would also require that their 'computer' be on 24/7, assuming the user wants to know the accurate time at any time of the day or night.
I don't think I implied such a requirement. Occasional synchronization should be sufficient, like existing radio clocks.
Bradley , "The library discussed in this list includes source code for one." Oh, that sounds interesting, but I'm not sure where to look. Are you able to give me link? "Occasional synchronization should be sufficient, like existing radio clocks." I was responding to your suggestion: "An even simpler design then would be to install software on the PC that would deliver the local time to your clock over the LAN. Get out of the TZ business altogether." With that scheme, as described, if the PC was not on when DST started/ended, the 'clock' would not be correct until the PC was on again. Regards, Daniel
On Thu, Nov 2, 2017 at 8:06 AM, Daniel Ford <dfnojunk@gmail.com> wrote:
"The library discussed in this list includes source code for one." Oh, that sounds interesting, but I'm not sure where to look. Are you able to give me link?
https://www.iana.org/time-zones https://github.com/eggert/tz
"Occasional synchronization should be sufficient, like existing radio clocks." I was responding to your suggestion: "An even simpler design then would be to install software on the PC that would deliver the local time to your clock over the LAN. Get out of the TZ business altogether." With that scheme, as described, if the PC was not on when DST started/ended, the 'clock' would not be correct until the PC was on again.
It would if the PC told the clock when/what the next UTC-offset transition was going to be.
Bradley, Thanks for the links, but they both appear to point to the standard TZdb code (which I've previously downloaded), not code specifically for Arduino. I doubt it would compile in the Arduino IDE (though I haven't yet tried). "It would if the PC told the clock when/what the next UTC-offset transition was going to be." Of course, but that contradicts your suggestion: "Get out of the TZ business altogether." So I'd be back to having to find a source of DST rules, and as mentioned, can't rely on any particular (unknown breed) local computer. Regards, Daniel
Zefram, Ah yes, I'd forgotten about the 'version' file; definitely the way to check for updates. Thanks for the clarification about the zone1970.tab contents. So maybe I can program the clock to make a once-only access to a TZdb mirror for that file, at first start-up, to fetch the region/cities list for the owner to choose from. If something ever changed to make that choice invalid, there could be a service routine in the firmware that allowed the user to re-do this once-only download for a new region/city selection. (Whew, things are getting complicated! :-) But the most exciting part of your e-mail was the mention of the IETF working group! I went there and found that they had concluded their work and issued RFC7808. I downloaded that and read it, and think that the "expand" Action is just what I'm looking for! So all I need to do is find one or more TZDIST servers that look like they should be around for the long-term. To my (simple) way of thinking, it would make sense if (some) NTP servers added this TZDIST functionality, so they became a 'one-stop-shop' for current time information, but whether they have/will is another matter. I'll start looking. Thanks again for the tip-off. Regards, Daniel
On 2017-11-02 19:54, Daniel Ford wrote:
Zefram,
Ah yes, I'd forgotten about the 'version' file; definitely the way to check for updates.
Thanks for the clarification about the zone1970.tab contents. So maybe I can program the clock to make a once-only access to a TZdb mirror for that file, at first start-up, to fetch the region/cities list for the owner to choose from. If something ever changed to make that choice invalid, there could be a service routine in the firmware that allowed the user to re-do this once-only download for a new region/city selection. (Whew, things are getting complicated! :-)
But the most exciting part of your e-mail was the mention of the IETF working group! I went there and found that they had concluded their work and issued RFC7808. I downloaded that and read it, and think that the "expand" Action is just what I'm looking for! So all I need to do is find one or more TZDIST servers that look like they should be around for the long-term.
Embedded devices must have some way of changing server names otherwise they risk becoming useless junk by the time they are built, or an annoying irritant to parts of the Internet, which may have legal, financial, social, and reputational risks. See https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Notable_cases
To my (simple) way of thinking, it would make sense if (some) NTP servers added this TZDIST functionality, so they became a 'one-stop-shop' for current time information, but whether they have/will is another matter. I'll start looking.
If the servers require TLS aka SSL, and maybe also DNSSEC, that may increase the requirements for an Arduino implementation to 32 bit AVR with lots of storage and Wifi shield with DNS, HTTP, HTTPS, SSL support. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
Woohoo! I've discovered (and downloaded) an Arduino library, written by Howard E. Hinnant: "This timezone library is a complete parser of the IANA timezone database. It provides for an easy way to access all of the data in this database, using the types from "date.h" and <chrono>. The IANA database also includes data on leap seconds, and this library provides utilities to compute with that information as well." All I need to find now are established TZdb mirrors that keep their copy up-to-date, so I don't overload the master TZdb with queries. Can anyone point me to any, please? Regards, Daniel --------------- Daniel Ford (Gerroa, Australia)
OK, so now I've abandoned all hope of finding an Arduino library that does all the heavy lifting for me, I now need to return to my 'roll-your-own' approach. First, I'd like to revisit my 1-Nov question and Zefram's 2-Nov answer (was probably sent 1-Nov from his timezone): -----
I had naively assumed that there was a 1:1 correspondence between region names in zone1970.tab and file names containing zones/rules for those regions.<
There is not. There's an approximate correspondence for most regions, but notably there's no "pacific" file, with most (but not all) Pacific/ zones being in the "australasia" file. ----- So my question now is, how does one know which file to search for zone/rules for any given region/city from zone1970.tab? And please don't answer 'run such-and-such a program from the code area', since I'm not doing this on a desktop machine, but an embedded system without a standard operating system. I need to understand the mechanism/algorithm so I can code my own processes. Thanks. Regards, Daniel
Daniel Ford wrote:
So my question now is, how does one know which file to search for zone/rules for any given region/city from zone1970.tab?
One doesn't. Traditionally, one reads all the data source files. It's a bit tricky to determine automatically which files these are; the Makefile knows which files to feed to zic, but to get it to hand over that information requires executing the Makefile with a fully-functional make(1). A recent innovation makes it easier, though: the tzdb-*.tar.lz distro includes a tzdata.zi file, which is in the source format and includes all of the data. It's not actually a source file, not including things like comments, but if you want to automatically process the data from the source format then it's a sensible starting point.
And please don't answer 'run such-and-such a program from the code area',
It's not a given that there would be an answer that avoids that sort of process. Traditionally we haven't had much concern about direct users lacking a full-strength coding environment. -zefram
On 11/20/2017 04:54 AM, Daniel Ford wrote:
So my question now is, how does one know which file to search for zone/rules for any given region/city from zone1970.tab?
As zefram mentioned, your best bet might be to look at tzdata.zi, which contains all the data for all the entries in zone1970.tab. You can get the current released version here: https://www.iana.org/time-zones/repository/tzdb/tzdata.zi This file contains only ASCII text; it is designed to be small and compressible. If your network client supports "Content-Encoding: gzip" (most do, these days), the payload is currently 25,895 bytes on the network and is 106,378 bytes uncompressed. This is small enough that it should not be an undue burden on the master iana.org servers if your small number of clocks periodically poll for updates in the usual way (they can first get the Last-Modified: header to see whether they're up-to-date). Doing it this way would relieve you of the burden of maintaining your own time servers, though of course your customers would then be at the mercy of the IANA, which has not yet committed to supporting this URL for the indefinite future. (Your call. :-)
Paul Eggert <eggert@cs.ucla.edu> writes:
As zefram mentioned, your best bet might be to look at tzdata.zi, which contains all the data for all the entries in zone1970.tab. You can get the current released version here:
Perhaps veering a bit from the thread topic, but: would it be reasonable to request that that file contain some version-identifying info? Ideal from my standpoint would be a tz release number, eg # This zic input file is in the public domain. # Zone data as of tzdata2017c. R A 1916 o - Jun 14 23s 1 S R A 1916 1919 - O Sun>=1 23s 0 - R A 1917 o - Mar 24 23s 1 S but if that's hard, a date would do. The body of the file is sufficiently not-human-readable that it's hard to identify what you're dealing with if you don't already know. regards, tom lane
Tom Lane wrote:
would it be reasonable to request that that file contain some version-identifying info?
Although we've avoided putting a version number into the existing binary format due to compatibility concerns, adding a comment to tzdata.zi shouldn't be so constrained. How about the attached patch? It prepends the following comment line to tzdata.zi in the current development version: # version 2017c-18-ga5a40b4 Here, "2017c-18-ga5a40b4" has the same meaning as in the file 'version': release 2017c plus 18 updates, with a Git checksum of a5a40b4. If the working files have uncommitted changes, the version string has "-dirty" appended.
Paul Eggert <eggert@cs.ucla.edu> writes:
How about the attached patch? It prepends the following comment line to tzdata.zi in the current development version: # version 2017c-18-ga5a40b4 Here, "2017c-18-ga5a40b4" has the same meaning as in the file 'version': release 2017c plus 18 updates, with a Git checksum of a5a40b4. If the working files have uncommitted changes, the version string has "-dirty" appended.
This answers my concern nicely. Thanks! regards, tom lane
participants (17)
-
Arthur David Olson -
Bradley White -
Brian Inglis -
Brian Inglis -
Daniel Ford -
David Patte ₯ -
Matt Johnson -
Michael Douglass -
Paul Eggert -
Paul.Koning＠dell.com -
Random832 -
Robert Elz -
Sanjeev Gupta -
SM -
Tim Parenti -
Tom Lane -
Zefram