Author: Gregory Maxwell 2012-10-10 00:03:52
Published on: 2012-10-10T00:03:52+00:00
In a discussion about software testing on the Bitcoin development mailing list, Jeff Garzik suggests writing data-driven tests that are compatible with clients other than the reference client. He also recommends embedding tests in testnet3 chain for better testing. The conversation then shifts to the problem of confirmation bias in software testing and how it leads to over-testing success cases and under-testing failure cases. Failure cases are critical for testing implementation compatibility and preventing network splits. Therefore, it is essential to have a diverse collection of testers who can think outside the box and test corner cases. To ensure comprehensive testing, developers should use multiple approaches rather than focusing on a single one. Combining different mechanisms of software testing is synergistic and enhances detection of issues. For example, almost all testing is improved by using valgrind, which lowers the threshold of issue detection significantly, and automatable testing (coded data driven, unit, and fuzz testing) works well with diverse compilation. However, valgrind is not available on Windows, and Dr. Memory may be an alternative, or ASAN could be ported to GCC to enable mingw ASAN builds.
Updated on: 2023-05-19T15:53:44.157449+00:00