Short Course 4, Part 1: Combinatorial Testing for AI Enabled Systems

Speakers Jaganmohan Chandrasekaran, Virgina Tech Erin Lanus, Virginia Tech, Research Assistant Professor Brian Lee, VTNSI M S Raunak, National Institute of Standards and Technology (NIST), Computer Scientist Rick Kuhn, National Institute of Standards & Technology, Computer Scientist Raghu Kacker, National Institute of Standards and Technology, Scientist Abstract: Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.