[YBA] TLD make using mawk causes malloc failure
Dear colleagues, I noticed that on all tz versions up to 2021e running make in the top-level directory results in a malloc failure on Debian 10 which uses the mawk alternate for awk: cc -DTZDIR='"/usr/share/zoneinfo"' -c -o difftime.o difftime.c rm -f libtz.a ar -rc libtz.a localtime.o asctime.o difftime.o strftime.o : libtz.a awk -v DATAFORM=`expr main.zi : '\(.*\).zi'` -f ziguard.awk \ africa antarctica asia australasia europe northamerica southamerica etcetera factory backward >main.zi.out malloc(): unsorted double linked list corrupted Aborted make: *** [Makefile:612: main.zi] Error 134 This can be corrected by installing gawk and running update-alternatives --config awk to select gawk as the awk alternate. I verified that the make completes without error using either GNU Awk 4.1.4, or GNU Awk 4.2.1. I did not see any warning about this in any of the NEW or README files, which is unfortunate as I spent several hours of my employers time understanding the problem and I suspect that others have seen this already. I suggest adding an INSTALL readme to say which environments are known to work and if there are specific know issues such as mawk. Best regards, - yba -- 9590 8E58 D30D 1660 C349 673D B205 4FC4 B8F5 B7F9 ~. .~ Tk Open Systems =}-------- Jonathan Ben-Avraham ("yba") ----------ooO--U--Ooo------------{= mailto:yba@tkos.co.il tel:+972.52.486.3386 http://tkos.co.il skype:benavrhm
On 19/01/2022 10:12, Jonathan Ben Avraham via tz wrote:
Dear colleagues, I noticed that on all tz versions up to 2021e running make in the top-level directory results in a malloc failure on Debian 10 which uses the mawk alternate for awk:
cc -DTZDIR='"/usr/share/zoneinfo"' -c -o difftime.o difftime.c rm -f libtz.a ar -rc libtz.a localtime.o asctime.o difftime.o strftime.o : libtz.a awk -v DATAFORM=`expr main.zi : '\(.*\).zi'` -f ziguard.awk \ africa antarctica asia australasia europe northamerica southamerica etcetera factory backward >main.zi.out malloc(): unsorted double linked list corrupted Aborted make: *** [Makefile:612: main.zi] Error 134
FWIW, in Debian 11 they switched to a different development stream of mawk. Debian 10 was using the Mark Brennan's mawk-1.3.3 with Debian patches (currently mawk-1.3.3-17, Debian maintainer Steve Langasek). Debian 11 uses Thomas Dickey's mawk-1.3.4 plus snapshots (currently mawk-1.3.4.20200120, Debian package version mawk-1.3.4.20200120-2, Debian maintainer Boyuan Yang). Debian's mawk-1.3.4.20200120-2 doesn't have the problem. The home page for Thomas Dickey's mawk 1.3.4 is: https://invisible-island.net/mawk/mawk.html He maintains it in RCS, but there are snapshots on GitHub at: https://github.com/ThomasDickey/mawk-snapshots I used git bisect (reversing the usual meanings of "good" and "bad") to determine that the problem was fixed by the 20100418 snapshot, tagged t20100418, commit: https://github.com/ThomasDickey/mawk-snapshots/commit/e23bdc528abf8e0752a3c2... The main changes for this snapshot were to the files "array.c", "array.w" ("array.c" is "untangle"d from the "Noweb" source in "array.w"), and "types.h".
This can be corrected by installing gawk and running
update-alternatives --config awk
to select gawk as the awk alternate. I verified that the make completes without error using either GNU Awk 4.1.4, or GNU Awk 4.2.1.
You could also build tz with make AWK=gawk (or edit the Makefile to change the AWK variable). -- -=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company )=- -=( registered in England & Wales. Regd. number: 02862268. )=- -=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=- -=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-
Hi Ian, Thanks for the information. A pity I had to go to this length to figure it all out. It should have been in the tz README or INSTALL documentation. - yba (Jonathan) On Wed, 19 Jan 2022, Ian Abbott wrote:
Date: Wed, 19 Jan 2022 14:00:22 +0000 From: Ian Abbott <abbotti@mev.co.uk> To: Jonathan Ben Avraham <yba@tkos.co.il>, tz@iana.org Subject: Re: [tz] [YBA] TLD make using mawk causes malloc failure
On 19/01/2022 10:12, Jonathan Ben Avraham via tz wrote:
Dear colleagues, I noticed that on all tz versions up to 2021e running make in the top-level directory results in a malloc failure on Debian 10 which uses the mawk alternate for awk:
cc -DTZDIR='"/usr/share/zoneinfo"' -c -o difftime.o difftime.c rm -f libtz.a ar -rc libtz.a localtime.o asctime.o difftime.o strftime.o : libtz.a awk -v DATAFORM=`expr main.zi : '\(.*\).zi'` -f ziguard.awk \ africa antarctica asia australasia europe northamerica southamerica etcetera factory backward >main.zi.out malloc(): unsorted double linked list corrupted Aborted make: *** [Makefile:612: main.zi] Error 134
FWIW, in Debian 11 they switched to a different development stream of mawk. Debian 10 was using the Mark Brennan's mawk-1.3.3 with Debian patches (currently mawk-1.3.3-17, Debian maintainer Steve Langasek). Debian 11 uses Thomas Dickey's mawk-1.3.4 plus snapshots (currently mawk-1.3.4.20200120, Debian package version mawk-1.3.4.20200120-2, Debian maintainer Boyuan Yang).
Debian's mawk-1.3.4.20200120-2 doesn't have the problem.
The home page for Thomas Dickey's mawk 1.3.4 is: https://invisible-island.net/mawk/mawk.html
He maintains it in RCS, but there are snapshots on GitHub at: https://github.com/ThomasDickey/mawk-snapshots
I used git bisect (reversing the usual meanings of "good" and "bad") to determine that the problem was fixed by the 20100418 snapshot, tagged t20100418, commit: https://github.com/ThomasDickey/mawk-snapshots/commit/e23bdc528abf8e0752a3c2...
The main changes for this snapshot were to the files "array.c", "array.w" ("array.c" is "untangle"d from the "Noweb" source in "array.w"), and "types.h".
This can be corrected by installing gawk and running
update-alternatives --config awk
to select gawk as the awk alternate. I verified that the make completes without error using either GNU Awk 4.1.4, or GNU Awk 4.2.1.
You could also build tz with
make AWK=gawk
(or edit the Makefile to change the AWK variable).
-- -=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company )=- -=( registered in England & Wales. Regd. number: 02862268. )=- -=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=- -=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-
-- 9590 8E58 D30D 1660 C349 673D B205 4FC4 B8F5 B7F9 ~. .~ Tk Open Systems =}-------- Jonathan Ben-Avraham ("yba") ----------ooO--U--Ooo------------{= mailto:yba@tkos.co.il tel:+972.52.486.3386 http://tkos.co.il skype:benavrhm
On 1/19/22 02:12, Jonathan Ben Avraham via tz wrote:
I did not see any warning about this in any of the NEW or README files, which is unfortunate
Yes it is. As far as I know you're the first person to mention this mawk bug to us. Thanks for doing that. Although the Makefile already warned that 'mawk' is glitchy, it mentioned only UTF-8 issues whereas this is more serious. I strengthened the wording by installing the attached proposed patch. In the past we have worked around some mawk bugs by simple changes to awk scripts to avoid the bugs[1][2]. I don't know how to do that easily here, though, as Ian's report makes it sound like the mawk bug is triggered by an awk array larger than 64k, and I expect it'd be a pain to rework the script to not use arrays that might exceed that size. If I'm wrong and there is a simple change to work around the mawk bug, please let me know. On older Debian it's a good idea anyway to run the command you mentioned (update-alternatives --config awk'), as in my experience mawk 1.3.3 is too buggy for many practical uses. [1]: https://github.com/eggert/tz/commit/464a055adfc9fc5282e8379563789943293113ac [2]: https://github.com/eggert/tz/commit/22875b5b069bb74b734dc7009e3dbadaff5a1904
participants (3)
-
Ian Abbott -
Jonathan Ben Avraham -
Paul Eggert