Software Testing & Verification Group Moonzoo Kim

SWTV group. CS Dept. KAIST Introduction of SWTV group 2/14 To Improve SW reliability through automated testing –Quality attribute for minimizing malfunctions of systems to reduces damage to human life or valuable properties (i.e., to fight software bug) Highly reliable SW technology is a key to the success of industrial products –The portion of SW in embedded devices increases continuously Home Network Intelligent Mobile Systems Highly Reliable Software-intensive Systems Intelligent Medical Devices Main Research Theme Home Service Robots

SWTV group. CS Dept. KAIST Introduction of SWTV group How to Assure Quality of SW? 3

SWTV group. CS Dept. KAIST Introduction of SWTV group Keynote by Bill OOPSLA, 2002 “… When you look at a big commercial software company like Microsoft, … We have as many testers as we have developers. Testers basically test all the time, and developers basically are involved in the testing process about half the time…” “… We're not in the software industry; we're in the testing industry, and writing the software is the thing that keeps us busy doing all that testing.” “…The test cases are unbelievably expensive; in fact, there's more lines of code in the test harness than there is in the program itself. Often that's a ratio of about three to one.” 4

SWTV group. CS Dept. KAIST Introduction of SWTV group Significance of Automated SW Analysis Software has become more ubiquitous and more complex at the same time Human resources are becoming less reliable and more expensive for highly complex software systems Computing resources are becoming ubiquitous and free China provides 10TB storage free –Amazon EC2 price: you can use thousands of 0.057$/hr for 3.2Ghz Quad-core CPU Remaining task? –To develop automated and scientific software analysis tools to utilize computing resource effectively and efficiently 5

SWTV group. CS Dept. KAIST Introduction of SWTV group Research Methodology SoftwareEngineering OK Counter example(s) or System modeling Requirement properties System spec. Automated Testing Req. spec. ProgrammingLanguages Algorithms EmbeddedSystems System programming SW Engineering Security Discrete math Algorithm PL SW TV group 6/14 Science Engineering assert( φ )

SWTV group. CS Dept. KAIST Introduction of SWTV group Ex. Testing a Triangle Decision Program Input : Read three integer values from the command line. The three values represent the length of the sides of a triangle. Output : Tell whether the triangle is 부등변삼각형, 이등변삼각형, 정삼각형 Create a Set of Test Cases for this program: (3,4,5), (2,2,1), (1,1,1) ? Condition 1: a > 0, b > 0, c > 0 Condition 2: a < b + c –Ex. (4, 2, 1) is an invalid triangle –Permutation of the above condition :a < b +c, b < a + c, c < a + b What if b + c exceeds 2 32 (i.e. overflow)? –long v.s. int v.s. short v.s. char Developers often fail to consider implicit preconditions –Cause of many hard-to-find bugs

SWTV group. CS Dept. KAIST Introduction of SWTV group 8/60 # of test cases required? ①4①4 ②10 ③50 ④100 # of feasible unique execution paths? 11 paths guess what test cases needed “Software Testing a craftsman’s approach” 2 nd ed by P.C.Jorgensen (no check for positive inputs)

SWTV group. CS Dept. KAIST Introduction of SWTV group Research Trends toward Quality Systems Academic research on embedded systems has reached stable stage –just adding new functionalities to a target system is not considered as an academic contribution anymore Research focus has moved to the quality of the systems from the mere functionalities of the systems –Energy efficient design, ez-maintenance, dynamic re-configuration, etc Software reliability is one of the highly pursued qualities –ASPLOS 2011 Best paper “S2E: a platform for in-vivo multi-path analysis for software EPFL –OSDI 2008 Best paper “Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Stanford –NSDI 2007 Best paper “Life, Death, and the Critical Transition: Finding Liveness Bugs in Systems U.C. San Diego 9/14

SWTV group. CS Dept. KAIST Introduction of SWTV group Collaborators of Our Group 10/14

뮤즈 (MUSE): 프로그램의 수많은 돌연변이들을 활용한버그 위치 추정 기법 Research SWTV SWTV group 11 Sequential program testing Concurrent program testing Fault localization Test cases Thread sched. scenarios Suspicious stmts.

SWTV group. CS Dept. KAIST Introduction of SWTV group 12

SWTV group. CS Dept. KAIST Introduction of SWTV group Concolic 테스팅 과제의 산업적 성취 년-2012년 2011년-2012년 2013년-2014년 2010년 산학 과제 수행 연도 Phase1(2010) Concolic testing 타당성 조사 Phase1(2010) Concolic testing 타당성 조사 Pilot 과제 적용을 통한 concolic testing 타당성 조사 오픈소스 소프트웨어 (busybox) 및 S 사 소프트웨어 (SLP(Tizen 전신 ) file manager, Samsung security library) 에 적용하여 새로운 버그 발견 Phase2( ) Concolic testing 도구 개발 Phase2( ) Concolic testing 도구 개발 S 사 개발 코드를 테스트하기 위한 자동화된 concolic unit testing 도구 CONBOL 개발 4MLOC 규모의 삼성 내장형 프로그램에 적용하여 결함 발견 및 개발자 보고 후 수정 기존 concolic testing 도구를 확장하여 bit 수준 정확도, BOF 오류 감지, 실행 속도 개선 수행 Phase3( ) Concolic testing 도구 속도 향상 및 오탐 감소 Phase3( ) Concolic testing 도구 속도 향상 및 오탐 감소 CONBOL 오탐 감소를 위한 unit-testing 전략, pre-condition 생성 기술 개발 CONBOL 실행 속도 및 사용성 개선 : 64bit 지원, 최신 SMT solver 지원 등 주 단위 적용을 통해 삼성 내장형 프로그램 및 open source project 의 실제 결함 검출

SWTV group. CS Dept. KAIST Introduction of SWTV group Concolic 테스팅 과제의 학문적 성취 년 2011년 2013년 2010년 2012년 2014년 산학 과제 수행 연도 FSE 2011 held in Szeged, Hungary 논문 발표 FSE: SE 분야 TOP 2 학회 ICSE 2012 held in Zurich, Switzerland, ICST 2012 held in Montreal Canada 논문 2 건 발표 ICSE: SE 분야 TOP 1 학회 ASE 2013 held in Palo Alto, CA, USA 논문 발표 삼성 논문상 동상 수상 ASE: SE 분야 TOP 3 학회

Recognition of Success of CONBOL at Samsung Electronics 15/23 Bronze Award at Samsung Best Paper Award Oct’s Best Practice Award Team leader Dr. Yoonkyu Jang received Samsung Award of Honor

SWTV group. CS Dept. KAIST Introduction of SWTV group Ph.D Students 16 Topic : Automated testing 2 top intl. journals such as IEEE TSE and FACJ 16 top intl. conf. Including ICSE/FSE/ASE Software system: –Scalable COncolic testing for Relibaility (SCORE) Topic: Concurrency testing J. of Systems and SW, JSTVR, ISSTA’12, ICST’13, ICST’14, etc. Samsung Humantech Bronze award Qualcomm fellowship award Software system: –COncurrent Bug dETection framework (COBET)

SWTV group. CS Dept. KAIST Introduction of SWTV group MS Students Topic : concurrent program testing and symbolic execution Best MS thesis award ICSE 2015 SEIP (top S rank), ICST ‘14 Best paper award at KCSE 2014 Best paper award at KIISE paper competition Qualcomm fellowship award Recent Alumni Topic : automated fault localization Attended ICSE India Preparing OOPSLA 2015 Topic : concolic testing

SWTV group. CS Dept. KAIST Introduction of SWTV group Questions? Comments? 18/14