Modernising firmware testing on POWER with Python and pre-release everything

Stewart Smith http://lca2018.linux.org.au/schedule/presentation/87/ Some people have the odd expectation that they will be able to (reliably) turn their computers on and off. On top of that, other people seem to have the entirely unreasonable expectation to have their computers reliably boot an Operating System, perform consistently, and not catch fire. In order to meet these outrageous expectations, we need to test firmware before we release it to an unsuspecting public. After all, isn't the gold standard of testing "if it boots, ship it" ? In the era of Blockchain, the gold standard no longer applies. As a firmware maintainer, I want the answer to a simple question: "Will I regret merging this code?". A simple and quick automated test suite is in order! Unfortunately, such things are not always around when you need them. In order to successfully deliver POWER9 systems (new processor, new operating systems, new machines, new BMC software), we would need a high quality, reliable, fast, easy to use, automated firmware test suite that could not only spot regressions, but find new bugs. This is a tale of code and organizational transformation and discovery during an incredibly tight and largely immobile release schedule. We'll cover: - Components of a modern POWER server and how one might test them (on purpose or by accident) - Practical use of Python in a mix of modern and legacy environments without going insane - (ab)using existing testing frameworks for (almost) fun and (hopefully) profit - How to (not) live with old enterprise distros - Transforming development, test, and management organizations, into test focused ones. - How on earth do you design a firmware compliance test before you have a test suite anyway? This talk is brought to you by: - A Perl script calling a perl script that parses XML with regex that ran a shell script to construct a python script to shell out to expect. - Unexpected bugs in the TTY layer - Soft Lockups - SOL Disconnected by BMC. This talk was given at Linux.conf.au 2018 (LCA2018) which was held on 22-26 January 2018 in Sydney Australia. linux.conf.au is a conference about the Linux operating system, and all aspects of the thriving ecosystem of Free and Open Source Software that has grown up around it. Run since 1999, in a different Australian or New Zealand city each year, by a team of local volunteers, LCA invites more than 500 people to learn from the people who shape the future of Open Source. For more information on the conference see https://linux.conf.au/ #linux.conf.au #linux #foss #opensource