1. CS453: Automated Software Testing
Moonzoo Kim
Software Testing and Verification Group
CS Dept. KAIST

2. Role of S/W: Increased in Everywhere
F-35 (8 Mloc)
2012년
자료출처: Watts Humphrey 2002

3. Safety Problems due to Poor Quality of SW

4. Static analysis falls short of detecting such complex bugs accurately
High false negatives
High false positives
Systematic and dynamic analysis (i.e. automated sw testing) is MUST for high quality SW
Moonzoo Kim

5. Current Practice for SW
-SW development cost takes 35% of total automotive production cost (Software Engineering for Automotive Systems Workshop, 2004)
-Due to complexity for reliable SW, the low productivity causes severe problem
SW developers have to follow systematic disciplines for building and analyzing software with high quality
This class focuses on the analysis activities

6. Software Development Cycle
A practical end-to-end formal framework for software development
A SW Development Framework for SW with High Assurance
Requirement
analysis
System
design
Design
analysis
Implement-
ation
Testing
Monitoring
Formal
require-
ment
Spec.
Formal
system
modeling
Model
analysis/
verification
Model-
assisted
code
generation
Model-
based
testing
Runtime
monitoring
and
checking

7. SW Development and Testing Model (a.k.a. V model)
Manual
Labor
Abstraction
Moonzoo Kim Provable SW Lab

8. Highly Reliable Systems
Main Target Systems
Embedded systems where highly reliable SW technology is a key to the success
The portion of SW in commercial embedded devices increases continuously
More than 50% of development time is spent on SW testing and debugging
Intelligent Medical Devices
Home Service Robots
-SW development cost takes 35% of total automotive production cost (Software Engineering for Automotive Systems Workshop, 2004)
-Due to complexity for reliable SW, the low productivity causes severe problem
Home Network
Intelligent Mobile Systems
Highly Reliable Systems

9. Strong IT Industry in South Korea
Time-to-Market?
SW Quality?
Not method oriented, but problem oriented research
Moonzoo Kim

10. Embedded Software in Two Different Domains
Conventional Testing
Concolic testing
Model checking
Consumer Electronics
Safety Critical Systems
Examples
Smartphones, flash memory platforms
Nuclear reactors, avionics, cars
Market competition
High
Low
Life cycle
Short
Long
Development time
Model-based development
None
Yes
Important value
Time-to-market
Safety
Not method oriented, but problem oriented research
Moonzoo Kim

11. How to Improve the Quality of SW
Systematic testing (can be still manual)
Coverage criteria
Mutation analysis
Testing through automated analysis tools
Scientific treatment of SW with computing power
Useful tools are available
Formal verification
Guarantee the absence of bugs

12. Questions??? Is automated testing really beneficial in industry?
Yes, dozens of success stories at Samsung
Is automated testing academically significant?
Yes, 3 Turing awardees in ‘07
Is automated testing too hard to learn and use?
No, there are tools available

13. S사 Concolic 테스팅 5년 과제 정리
목표: S사에서 개발한 임베디드 C 프로그램의 효과적인 버그 발견을 위해 최신 자동화 테스트 기법인 Concolic 테스팅 기술 확보
총 개발 기간: 41.5개월 (2010년 ~ 2014년)
총 개발 비용: 414 백 만원
성과:
도구 개발: S사의 임베디드 C 프로그램의 자동화된 버그 탐지 및 유닛 테스트를 위한 Concolic unit-testing 도구 CONBOL 개발
CONBOL 을 적용하여 S사 개발4MLOC 규모의 내장형 SW와 0.2MLOC 규모의 open-source 코드의 실제 버그 발견
논문 발표: ICSE 2012, FSE 2011 와 같은 SE 분야 TOP1, 2 학회를 비롯하여 ICST 2012, ASE 2013 등 소프트웨어 공학 분야 저명 학회에 총 4건의 논문 발표

14. Concolic testing 도구 속도 향상 및 오탐 감소
산학 과제
수행 연도
2011년-2012년
2013년-2014년
2010년
Phase1(2010)
Concolic testing
타당성 조사
Pilot 과제 적용을 통한 concolic testing 타당성 조사
오픈소스 소프트웨어(busybox) 및 S사 소프트웨어(SLP(Tizen 전신) file manager, Samsung security library) 에 적용하여 새로운 버그 발견
Phase2( )
Concolic testing 도구 개발
S사 개발 코드를 테스트하기 위한 자동화된 concolic unit testing 도구 CONBOL 개발
4MLOC 규모의 삼성 내장형 프로그램에 적용하여 결함 발견 및 개발자 보고 후 수정
기존 concolic testing 도구를 확장하여 bit 수준 정확도, BOF 오류 감지, 실행 속도 개선 수행
Phase3( )
Concolic testing 도구 속도 향상 및 오탐 감소
CONBOL 오탐 감소를 위한 unit-testing 전략, pre-condition 생성 기술 개발
CONBOL 실행 속도 및 사용성 개선: 64bit 지원, 최신 SMT solver 지원 등
주 단위 적용을 통해 삼성 내장형 프로그램 및 open source project의 실제 결함 검출

15. Concolic 테스팅 과제의 학문적 성취 산학 과제 수행 연도
2011년
2013년
2010년
2012년
2014년
FSE 2011 held in Szeged, Hungary 논문 발표
FSE: SE 분야 TOP 2 학회
ICSE 2012 held in Zurich, Switzerland, ICST 2012 held in Montreal Canada 논문 2건 발표
ICSE: SE 분야 TOP 1 학회
ASE 2013 held in Palo Alto, CA, USA 논문 발표
삼성 논문상 동상 수상
ASE: SE 분야 TOP 3 학회

16. Verification of High-Availability Protocol
We develop a formal model of high-availability protocol used in commercial security appliances
HA protocol coordinates a group of firewalls
We found several problems in HA
regarding a master election procedure
Master
Backup Master
Slave

17. Home Service Robot
Designed for providing various services to human user
- Service areas : home security, patient caring, cleaning, etc
SAIT started development of SHR00 from 2002
4 separate teams (13 persons)
Vision recognition, speech recognition, simultaneous localization and mapping (SLAM), actuator
Both SHR00 and SHR50 suffered feature interaction problems
SAIT decided to develop SHR100 from scratch
SAIT requested to improve the reliability of SHR100 in six months
SHR100 is written in 17K line of C/C++

18. OneNAND® Flash Memory Devices
Each memory cell can be written limited number of times only
XIP by emulating NOR interface through demand-paging scheme
Performance enhancement
Source:
Software Center
of Samsung
Electronics ‘06
App1
App2
App3
Demand
Paging
Manager (DPM)
File
System
Unified
Storage
Platform
Flash
Translation Layer OS
Adapt-
ation
Module
Sector
Translation (STL)
Block
Management (BML)
Low Level (LLD)
Device Driver
OneNAND® Flash Memory Devices

19. Smartphone SW Platform
We have developed CONcrete and symBOLic (CONBOL) framework that is an automated concolic unit testing tool based-on CREST-BV for embedded software
Target
source code for embedded platform
GCC compatible
source code
Porting
Module
Pre-processor
Module
Instru-mentor
Instru-mented code
GCC
Linux executable
Concolic Executor
Defect
Report
Test-generator
Module
Unit test
driver+stub code
SMT
Solver
CREST-BV extension
New module
External tool
Legend
Coverage
Report
Symbolic
Library

20. Research Trends toward Quality Systems
Academic research on developing embedded systems has reached stable stage
just adding a new function to a target system is not considered as an academic contribution anymore
Research focus has moved on to the quality of the systems from the mere functionalities of the systems
Energy efficient design, ez-maintenance, dynamic configuration, etc
Software reliability is one of the highly pursued qualities
USENIX Security 2015 best paper
“Under-Constrained Symbolic Execution: Correctness Checking for Real Stanford
ICSE 2014 best paper
“Enhancing Symbolic Execution with CMU
ASPLOS 2011 Best paper
“S2E: a platform for in-vivo multi-path analysis for software EPFL
OSDI 2008 Best paper
“Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Stanford
NSDI 2007 Best paper
“Life, Death, and the Critical Transition: Finding Liveness Bugs in Systems U.C. San Diego

21. Tool-based Interactive Learning
Code analyzer
C/C++ AST parser: Clang
Language independent Intermediate representation (IR) : LLVM
Model checker
Explicit model checker: Spin home page
Software model checker
Bounded model checker for C program: CBMC home page 
Satisfiability solver
MiniSAT home page
Satisfiability Module Solver
Yices home page
Z3 home page
Concolic testing tools
CREST home page

22. Class Schedule Part II: Logic model checking Part IV: Concolic testing
Week1: overview on automated SW analysis techniques
Part I: Coverage criteria and code manipulation
Week 2: graph coverage
Week 3: logic coverage
Week 4: clang/llvm tutorial
HW1: understanding of coverage criteria
HW2-3: branch coverage measuring tool
Part II: SW model checking
Week 5: SAT-based bounded model checking
Week 6: SW model checking case study
Week 7: SAT solver
Week 8: mid term exam
HW4: SW model checking to verify program
HW5: SW model checking to solve puzzles
Part II: Logic model checking
Part IV: Concolic testing

23. Final Remarks For undergraduate students:
Highly recommend URP studies or independent studies
Ex. 이준희 (05학번) got a silver award and macbook air notebook 
Debugging Linux kernel through model checking to detect concurrency bugs
Ex2. Nam Dang wrote down a paper on distributed concolic testing
Y.Kim, M.Kim, N.Dang, Scalable Distributed Concolic Testing: a Case Study on a Flash Storage Platform, Verified Software Intl. Conf. on Theoretical Aspects of Computing (ICTAC), Aug 2010

24. Final Remarks For graduate students:
Welcome research discussions to apply formal analysis techniques
Systematically testing/debugging C programs
Concurrency bug detection
Model-based testing
Pre-requisite:
Knowledge of the C/C++/Java programing language
Basic understanding of linux/unix
~6 hours of analysis/programming per week for HW