CS453: Automated Software Testing Moonzoo Kim Software Testing and Verification Group CS Dept. KAIST

Role of S/W: Increased in Everywhere F-35 (8 Mloc) 2012년 자료출처: Watts Humphrey 2002

Safety Problems due to Poor Quality of SW

Static analysis falls short of detecting such complex bugs accurately High false negatives High false positives Systematic and dynamic analysis (i.e. automated sw testing) is MUST for high quality SW Moonzoo Kim

Current Practice for SW -SW development cost takes 35% of total automotive production cost (Software Engineering for Automotive Systems Workshop, 2004) -Due to complexity for reliable SW, the low productivity causes severe problem SW developers have to follow systematic disciplines for building and analyzing software with high quality This class focuses on the analysis activities

Software Development Cycle A practical end-to-end formal framework for software development A SW Development Framework for SW with High Assurance Requirement analysis System design Design analysis Implement- ation Testing Monitoring Formal require- ment Spec. Formal system modeling Model analysis/ verification Model- assisted code generation Model- based testing Runtime monitoring and checking

SW Development and Testing Model (a.k.a. V model) Manual Labor Abstraction Moonzoo Kim Provable SW Lab

Highly Reliable Systems Main Target Systems Embedded systems where highly reliable SW technology is a key to the success The portion of SW in commercial embedded devices increases continuously More than 50% of development time is spent on SW testing and debugging Intelligent Medical Devices Home Service Robots -SW development cost takes 35% of total automotive production cost (Software Engineering for Automotive Systems Workshop, 2004) -Due to complexity for reliable SW, the low productivity causes severe problem Home Network Intelligent Mobile Systems Highly Reliable Systems

Strong IT Industry in South Korea Time-to-Market? SW Quality? Not method oriented, but problem oriented research Moonzoo Kim

Embedded Software in Two Different Domains Conventional Testing Concolic testing Model checking Consumer Electronics Safety Critical Systems Examples Smartphones, flash memory platforms Nuclear reactors, avionics, cars Market competition High Low Life cycle Short Long Development time Model-based development None Yes Important value Time-to-market Safety Not method oriented, but problem oriented research Moonzoo Kim

How to Improve the Quality of SW Systematic testing (can be still manual) Coverage criteria Mutation analysis Testing through automated analysis tools Scientific treatment of SW with computing power Useful tools are available Formal verification Guarantee the absence of bugs

Questions??? Is automated testing really beneficial in industry? Yes, dozens of success stories at Samsung Is automated testing academically significant? Yes, 3 Turing awardees in ‘07 Is automated testing too hard to learn and use? No, there are tools available

S사 Concolic 테스팅 5년 과제 정리 목표: S사에서 개발한 임베디드 C 프로그램의 효과적인 버그 발견을 위해 최신 자동화 테스트 기법인 Concolic 테스팅 기술 확보 총 개발 기간: 41.5개월 (2010년 ~ 2014년) 총 개발 비용: 414 백 만원 성과: 도구 개발: S사의 임베디드 C 프로그램의 자동화된 버그 탐지 및 유닛 테스트를 위한 Concolic unit-testing 도구 CONBOL 개발 CONBOL 을 적용하여 S사 개발4MLOC 규모의 내장형 SW와 0.2MLOC 규모의 open-source 코드의 실제 버그 발견 논문 발표: ICSE 2012, FSE 2011 와 같은 SE 분야 TOP1, 2 학회를 비롯하여 ICST 2012, ASE 2013 등 소프트웨어 공학 분야 저명 학회에 총 4건의 논문 발표 2019-01-12

Concolic testing 도구 속도 향상 및 오탐 감소 산학 과제 수행 연도 2011년-2012년 2013년-2014년 2010년 Phase1(2010) Concolic testing 타당성 조사 Pilot 과제 적용을 통한 concolic testing 타당성 조사 오픈소스 소프트웨어(busybox) 및 S사 소프트웨어(SLP(Tizen 전신) file manager, Samsung security library) 에 적용하여 새로운 버그 발견 Phase2(2011-2012) Concolic testing 도구 개발 S사 개발 코드를 테스트하기 위한 자동화된 concolic unit testing 도구 CONBOL 개발 4MLOC 규모의 삼성 내장형 프로그램에 적용하여 결함 발견 및 개발자 보고 후 수정 기존 concolic testing 도구를 확장하여 bit 수준 정확도, BOF 오류 감지, 실행 속도 개선 수행 Phase3(2013-2014) Concolic testing 도구 속도 향상 및 오탐 감소 CONBOL 오탐 감소를 위한 unit-testing 전략, pre-condition 생성 기술 개발 CONBOL 실행 속도 및 사용성 개선: 64bit 지원, 최신 SMT solver 지원 등 주 단위 적용을 통해 삼성 내장형 프로그램 및 open source project의 실제 결함 검출 2019-01-12

Concolic 테스팅 과제의 학문적 성취 산학 과제 수행 연도 2011년 2013년 2010년 2012년 2014년 FSE 2011 held in Szeged, Hungary 논문 발표 FSE: SE 분야 TOP 2 학회 ICSE 2012 held in Zurich, Switzerland, ICST 2012 held in Montreal Canada 논문 2건 발표 ICSE: SE 분야 TOP 1 학회 ASE 2013 held in Palo Alto, CA, USA 논문 발표 삼성 논문상 동상 수상 ASE: SE 분야 TOP 3 학회 2019-01-12

Verification of High-Availability Protocol We develop a formal model of high-availability protocol used in commercial security appliances HA protocol coordinates a group of firewalls We found several problems in HA regarding a master election procedure Master Backup Master Slave

Home Service Robot Designed for providing various services to human user - Service areas : home security, patient caring, cleaning, etc SAIT started development of SHR00 from 2002 4 separate teams (13 persons) Vision recognition, speech recognition, simultaneous localization and mapping (SLAM), actuator Both SHR00 and SHR50 suffered feature interaction problems SAIT decided to develop SHR100 from scratch SAIT requested to improve the reliability of SHR100 in six months SHR100 is written in 17K line of C/C++

OneNAND® Flash Memory Devices Each memory cell can be written limited number of times only XIP by emulating NOR interface through demand-paging scheme Performance enhancement Source: Software Center of Samsung Electronics ‘06 App1 App2 App3 Demand Paging Manager (DPM) File System Unified Storage Platform Flash Translation Layer OS Adapt- ation Module Sector Translation (STL) Block Management (BML) Low Level (LLD) Device Driver OneNAND® Flash Memory Devices

Smartphone SW Platform We have developed CONcrete and symBOLic (CONBOL) framework that is an automated concolic unit testing tool based-on CREST-BV for embedded software Target source code for embedded platform GCC compatible source code Porting Module Pre-processor Module Instru-mentor Instru-mented code GCC Linux executable Concolic Executor Defect Report Test-generator Module Unit test driver+stub code SMT Solver CREST-BV extension New module External tool Legend Coverage Report Symbolic Library

Research Trends toward Quality Systems Academic research on developing embedded systems has reached stable stage just adding a new function to a target system is not considered as an academic contribution anymore Research focus has moved on to the quality of the systems from the mere functionalities of the systems Energy efficient design, ez-maintenance, dynamic configuration, etc Software reliability is one of the highly pursued qualities USENIX Security 2015 best paper “Under-Constrained Symbolic Execution: Correctness Checking for Real Code” @ Stanford ICSE 2014 best paper “Enhancing Symbolic Execution with Veritesting” @ CMU ASPLOS 2011 Best paper “S2E: a platform for in-vivo multi-path analysis for software systems” @ EPFL OSDI 2008 Best paper “Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs” @ Stanford NSDI 2007 Best paper “Life, Death, and the Critical Transition: Finding Liveness Bugs in Systems Code” @ U.C. San Diego

Tool-based Interactive Learning Code analyzer C/C++ AST parser: Clang Language independent Intermediate representation (IR) : LLVM Model checker Explicit model checker: Spin home page Software model checker Bounded model checker for C program: CBMC home page  Satisfiability solver MiniSAT home page Satisfiability Module Solver Yices home page Z3 home page Concolic testing tools CREST home page

Class Schedule Part II: Logic model checking Part IV: Concolic testing Week1: overview on automated SW analysis techniques Part I: Coverage criteria and code manipulation Week 2: graph coverage Week 3: logic coverage Week 4: clang/llvm tutorial HW1: understanding of coverage criteria HW2-3: branch coverage measuring tool Part II: SW model checking Week 5: SAT-based bounded model checking Week 6: SW model checking case study Week 7: SAT solver Week 8: mid term exam HW4: SW model checking to verify program HW5: SW model checking to solve puzzles Part II: Logic model checking Part IV: Concolic testing

Final Remarks For undergraduate students: Highly recommend URP studies or independent studies Ex. 이준희 (05학번) got a silver award and macbook air notebook  Debugging Linux kernel through model checking to detect concurrency bugs Ex2. Nam Dang wrote down a paper on distributed concolic testing Y.Kim, M.Kim, N.Dang, Scalable Distributed Concolic Testing: a Case Study on a Flash Storage Platform, Verified Software Track @ Intl. Conf. on Theoretical Aspects of Computing (ICTAC), Aug 2010

Final Remarks For graduate students: Welcome research discussions to apply formal analysis techniques Systematically testing/debugging C programs Concurrency bug detection Model-based testing Pre-requisite: Knowledge of the C/C++/Java programing language Basic understanding of linux/unix ~6 hours of analysis/programming per week for HW