On 2014-08-08 17:21, Lester Caine wrote:
That I think is a nice summary Tim
All I would add is perhaps a reference to the amount of work required to fix the regressions in testing third party interfaces like PHP and Java and other platforms. It is the fact that we can not gauge if any of the changes do affect users that is the problem. That someone now gets a different answer in a grey area of the data may well not be important, but not knowing that a change HAS happened means that the effect of a change can not be reviewed? And perhaps the very evidence we are looking for discovered ...
Perhaps some comments on the scale and effort required for commercial/professional regression test suites and SOPs would be appropriate. I would expect a regression test suite to run zic to list all transitions in all zones, diff that output against the baseline output, look for release notes explaining why each change has occurred, document that; follow up any unexplained diffs with the maintainer, and document that, then pass the diff and documentation to the product release manager, for review and reappraisal or approval to promote the change to a production release. If someone could contribute such a regression test script and baseline output, it would allow the maintainer to better evaluate the potential impact of any proposed changes on products using the package. I also agree that normal revision control practice is to create a (tagged) branch for each potential change, review the diffs against the trunk, make any other changes indicated and add to the branch, then add the branch to the proposed changes expected to be merged into the trunk in the next release. On release, add the release tag to everything in the trunk. This also has the benefit that downstream consumers of the packages can more easily create their own forks which could exclude some changes or include their own patches. Each branch change would also normally include the output of the regression test suite run against those changes and checked in with the code. -- Take care. Thanks, Brian Inglis