I've been doing some surgery on localtime.c (perhaps more on that later), and to check my work I felt the need for a decent gmtime/localtime/mktime test suite. I'm sure such a thing has been written numbers of times before, and there may be some reason not to have one in the tzcode distribution, but if not, I'd like to donate this one for inclusion. It runs in two modes, one driven by a script containing (in effect) a list of known-good (time_t, struct tm) pairs, and another which exhaustively runs a potentially very large set of synthesized tests over a specified range. The script contains lines such as gmtime 1234567890 2009-02-13 23:31:30 and tz America/New_York localtime 483811200 1985-05-01 12:00:00 z=EDT o=-14400 dst=1 which are used to drive reciprocal pairs of tests on gmtime/timegm or localtime/mktime, in a reasonably obvious way. A line like the first one tests both that gmtime(1234567890) gives 2009-02-13, and also that timegm on 2009-02-13 23:31:30 gives 1234567890. There are keywords for setting the expected output values for zone, offset, and DST flag (as shown). There are also additional keywords to set the expected values of tm_wday and tm_yday. (See the attached test script for more examples.) I'm also enclosing a patch to the tzcode Makefile to add a "make tests" target, building and appropriately invoking this tool. At the moment this tool tests only localtime, gmtime, mktime, and timegm. It would make sense to have coverage of ctime and strftime, too, which would be straightforward enough to add. Note that the zone names in the attached 'testcases' script match those built by the stock tzdata distribution, but not necessarily those installed on the machine where you may be trying to run this. See the long comment at the top of testsuite.c for additional information. Enjoy! Steve Summit scs@eskimo.com
Thanks for doing that! It should be quite helpful. I briefly looked into it and have some comments: * Do you use git? If so, the output of 'git format-patch' makes it easier to review proposed changes. The first attached patch attempts to be identical to the patch you sent, except in 'git format-patch' format. * The second attached patch cleans up a few minor problems I noticed when building the test suite: ** "make tests" assumed "." was in PATH. ** "make clean" didn't clean up the test suite. ** Missing NEWS item. ** "make CFLAGS='$(GCC_DEBUG_FLAGS)'" complained about a few missing attributes and unused variables and so forth. ** TM_ZONE and TM_GMTOFF should be consistent with private.h. ** Symbols that needn't be extern should be static. ** Global variables that aren't modified should be const. ** I prefer "const" after the type it modifies, for consistency with types like "int * const *". ** I prefer a function's return type to be on the previous line, so that there is room for decorations like "static". Other comments: * The test suite fails on my platform (Fedora 23 x86-64). Why would that be? The first few lines of error diagnostics are: ./testsuite testcases testcases, line 1598: localtime: expected 1969-12-31 19:00:00, got 1969-12-31 20:00:00 testcases, line 1598: localtime: inverse expected 0, got -3600 testcases, line 1599: localtime: expected 1901-12-13 15:45:52, got 1901-12-13 16:45:52 testcases, line 1599: localtime: inverse expected -2147483648, got -2147487248 testcases, line 1604: localtime: expected 1969-12-31 16:00:00, got 1969-12-31 17:00:00 ... * It should be easier to do the tests with "make tests". Users shouldn't need to modify the makefile; it should automatically do the 64-bit tests on all platforms, making sure that they do the right thing on 32-bit hosts. We can put the expensive tests in a separate target "make tests-expensive". Perhaps we should fold this into "make check" (with a new rule "make check-expensive") to simplify things further. * The snprintfs should be fixed to never do silent truncation on any platform. The second one can use INT_STRLEN_MAXIMUM to do that. * The signal handler has undefined behavior. It should just write (not printf) to stderr and then _exit (not exit). * Would you mind if we changed "if(" to "if (", and similarly for other keywords, for consistency with the other tz code?
No problem with most of those cosmetic details, I knew there'd be some massaging necessary to conform to the "house style". Somehow I didn't even notice that the existing Makefile hade a 'check' target! Yes, it makes sense to fold this suite in there, and moving the expensive tests to a separate target is a good idea, too. I'm most concerned, of course, about the tests that failed for you. Now, 19:00 really is the right value for time 0 in EST; DST was not active in December in 1970! But I think I remember seeing weird failures like this, myself, which I at first attributed to a mismatch between the just-compiled version of localtime.c I was running versus the somewhat older tzdata files already installed on my system. But I think later I discovered it was because I wasn't actually hitting *any* zoneinfo files, because the stock Makefile's TZDIR didn't point to where my system's were. So that's the first thing I'd check. (Me, I ended up first pointing TZDIR at /usr/share which is where my system's files were, then later at a freshly-fetched and zicced copy of tzdata2015g.) And if that's it, it means that two out of the first two users have had this same problem, meaning that more will, too, so it'd be worth doing something to forestall it. Perhaps the test suite could print an explicit error message if it detects that valid zoneinfo files aren't being found. (Come to think of it, I had already been planning to add some kind of an optionally noisy warning to localtime.c for that case; thanks for the reminder!) Paul wrote:
Thanks for doing that! It should be quite helpful. I briefly looked into it and have some comments:
* Do you use git? If so, the output of 'git format-patch' makes it easier to review proposed changes. The first attached patch attempts to be identical to the patch you sent, except in 'git format-patch' format.
* The second attached patch cleans up a few minor problems I noticed when building the test suite:
** "make tests" assumed "." was in PATH.
** "make clean" didn't clean up the test suite.
** Missing NEWS item.
** "make CFLAGS='$(GCC_DEBUG_FLAGS)'" complained about a few missing attributes and unused variables and so forth.
** TM_ZONE and TM_GMTOFF should be consistent with private.h.
** Symbols that needn't be extern should be static.
** Global variables that aren't modified should be const.
** I prefer "const" after the type it modifies, for consistency with types like "int * const *".
** I prefer a function's return type to be on the previous line, so that there is room for decorations like "static".
Other comments:
* The test suite fails on my platform (Fedora 23 x86-64). Why would that be? The first few lines of error diagnostics are:
./testsuite testcases testcases, line 1598: localtime: expected 1969-12-31 19:00:00, got 1969-12-31 20:00:00 testcases, line 1598: localtime: inverse expected 0, got -3600 testcases, line 1599: localtime: expected 1901-12-13 15:45:52, got 1901-12-13 16:45:52 testcases, line 1599: localtime: inverse expected -2147483648, got -2147487248 testcases, line 1604: localtime: expected 1969-12-31 16:00:00, got 1969-12-31 17:00:00 ...
* It should be easier to do the tests with "make tests". Users shouldn't need to modify the makefile; it should automatically do the 64-bit tests on all platforms, making sure that they do the right thing on 32-bit hosts. We can put the expensive tests in a separate target "make tests-expensive". Perhaps we should fold this into "make check" (with a new rule "make check-expensive") to simplify things further.
* The snprintfs should be fixed to never do silent truncation on any platform. The second one can use INT_STRLEN_MAXIMUM to do that.
* The signal handler has undefined behavior. It should just write (not printf) to stderr and then _exit (not exit).
* Would you mind if we changed "if(" to "if (", and similarly for other keywords, for consistency with the other tz code?
Steve Summit wrote:
I think later I discovered it was because I wasn't actually hitting *any* zoneinfo files,
Bingo. After doing a 'make install', 'make tests' seems to work. People should be able to run tests without installing, though; indeed that should be the expected and default way to test. I'll take a crack at if it I find the time. Here's what the output of 'make tests' looks like on my machine: ./testsuite testcases scripted tests succeeded; trying exhaustive tests (this may take a minute or two) ./testsuite -x -f 1901-12-14 -t 2038-01-19 -i 1753 exhaustive tests: from 1901-12-14 to 2038-01-19 by 1753 testsuite: no or GMT/UTC time zone set, so no nontrivial tz offset tested try setting TZ or invoking with -s 2450054 tests completed all tests completed successfully I'm not sure what that warning about "no nontrivial tz offset tested" means, but I hope we can fix the testsuite to set TZ itself if that's what it needs to do. The tests took 38 seconds on my 6-year-old work desktop, which is a tad slow for my taste, but we can move the expensive tests to 'make check-expensive' or perhaps speed up the library and/or tester if we can find the time....
Replies to additional points in three of Paul's messages:
It should be easier to do the tests with "make tests". Users shouldn't need to modify the makefile;
I agree; I'm just not sure how to.
it should automatically do the 64-bit tests on all platforms, making sure that they do the right thing on 32-bit hosts.
(Is "the right thing" to just ignore them?)
testsuite.c assumes time_t is equivalent to long, by using the %ld printf format to print it. That's not portable. time_t might have a different width from long, and time_t might be unsigned. zdump.c's tformat function gets it right.
That was indeed a bit of carelessness on my part, that I needed to clean up. I'll take a look at zdump's technique.
I think later I discovered it was because I wasn't actually hitting *any* zoneinfo files,
(I am making progress on better error reporting for missing zoneinfo files; more on that later.)
Bingo. After doing a 'make install', 'make tests' seems to work. People should be able to run tests without installing, though; indeed that should be the expected and default way to test.
Again I agree but am not sure how to arrange it. The issue is that tzcode and tzdata are separate packages; I'm not sure what the expectation is about how and where people will download them with respect to each other. And what if someone downloads and tries to build tzcode without tzdata at all? Perhaps tzcode would have to contain its own miniature, stripped-down subset of tzdata, just for testing.
I'm not sure what that warning about "no nontrivial tz offset tested" means, but I hope we can fix the testsuite to set TZ itself if that's what it needs to do.
Setting TZ is what I needed to do, although I'm quite surprised that you had the same problem, so this will bear further investigation. My issue (under MacOS) is that although TZ is not set when I log in, the libc versions of localtime() et al. somehow do know what zone I'm in. But tzcode's localtime.c does not. I assumed Apple had tweaked tzset() to fetch the default zone from their own special place. But I think you said you're on Fedora, so where's it storing its default zone, that tzdata's tzset() can't find?
The tests took 38 seconds on my 6-year-old work desktop,
Not bad. I tuned them to take a minute on my several-year-old MacBook.
which is a tad slow for my taste, but we can move the expensive tests to 'make check-expensive' or perhaps speed up the library...
In fact I have a 3x speedup waiting in the wings, which I was waiting to post until the dust died down from this thread. :-)
participants (2)
-
Paul Eggert -
scs@eskimo.com