Presentation is loading. Please wait.

Presentation is loading. Please wait.

Shadow Shadow of a Doubt: Testing for Divergences Between Software Versions Hristina PalikarevaTomasz KuchtaCristian Cadar ICSE’16, 20 th May 2016 This.

Published byJordan Atkinson Modified over 8 years ago

Similar presentations

Presentation on theme: "Shadow Shadow of a Doubt: Testing for Divergences Between Software Versions Hristina PalikarevaTomasz KuchtaCristian Cadar ICSE’16, 20 th May 2016 This."— Presentation transcript:

1 Shadow Shadow of a Doubt: Testing for Divergences Between Software Versions Hristina PalikarevaTomasz KuchtaCristian Cadar ICSE’16, 20 th May 2016 This work is supported by EPSRC and Microsoft Research

2  Software patches  Frequent, at the core of software evolution  New features, bug fixes, better performance, usability  Poorly tested in practice  May introduce bugs 2 Motivation

3 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } Motivation: an example 3 Old

4 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }  Test cases: x = 0, x = 100, x = 101 Motivation: an example 4 OldNew 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

5  Test cases: x = 0, x = 100, x = 101, 100% code coverage Motivation: an example 5 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } OldNew

6  Test cases: x = 0, x = 100, x = 101, 100% code coverage  Only 50% new behaviour coverage  Code coverage not sufficient! Motivation: an example 6 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } OldNew

7  Shadow  Shadow symbolic execution technique  Focuses on the new behaviours of the patch  Technique for unifying two versions of a program  Execute in a single symbolic execution instance  A patch testing approach  Shadow symbolic execution  Enhanced cross-version checks 7 Contributions

8 8 Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

9 9 x+1 > 100 x+1 ≤ 100 Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

10 10 x+1 > 100 x+1 ≤ 100 Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

11 11 x+1 > 100 x+1 ≤ 100 Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

12 12 x+1 > 100 x+1 ≤ 100 Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

13 x + 1 ≤ 100 13 x + 1 > 100 x+1 ≤ 100 Symbolic execution x is a symbolic variable 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

14 x + 1 > 100 14 e.g. x = 1000 e.g. x = 0 x is a symbolic variable x+1 > 100 x+1 ≤ 100 Symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } x + 1 ≤ 100

15 Shadow Shadow symbolic execution

16  Old and new version in the same instance  The two versions are combined  Executed in lock-step fashion  The old version shadows the new one  Focus on the new behaviour  Versions take different sides of a branch 16 Shadow Shadow symbolic execution

17 01 int gt_100(unsigned x) { 02 unsigned y = x + 1; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 17 Old New Shadow Shadow symbolic execution

18 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 18 Combined Shadow Shadow symbolic execution

19 19 Shadow Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

20 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 20 Shadow Shadow symbolic execution

21 21 4-way fork Shadow Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

22 22 x+1 ≤ 100 x+1 > 100 Shadow Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

23 23 x ≤ 100 x > 100 x+1 ≤ 100 x+1 > 100 x ≤ 100 x > 100 Shadow Shadow symbolic execution 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

24 24 new:else old:else new:else old:else new:else old:then new:else old:then new:then old:else new:then old:else new:then old:then new:then old:then x+1 ≤ 100 x+1 > 100 Shadow Shadow symbolic execution x ≤ 100 x > 100 x ≤ 100 x > 100 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 }

25 25 x+1 ≤ 100 x+1 > 100 Shadow Shadow symbolic execution x ≤ 100 x > 100 x ≤ 100 x > 100 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } new:else old:else new:else old:else new:else old:then new:else old:then new:then old:else new:then old:else new:then old:then new:then old:then

26 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } 26 100 ✓ x+1 ≤ 100 x+1 > 100 Shadow Shadow symbolic execution x ≤ 100 x > 100 x ≤ 100 x > 100 new:else old:else new:else old:else new:else old:then new:else old:then new:then old:else new:then old:else new:then old:then new:then old:then

27 27 Max Int x+1 ≤ 100 x+1 > 100 Shadow Shadow symbolic execution x ≤ 100 x > 100 x ≤ 100 x > 100 01 int gt_100(unsigned x) { 02 unsigned y = change(x, x + 1); 03 if (y > 100) 04 return 1; 05 else 06 return 0; 07 } new:else old:else new:else old:else new:else old:then new:else old:then new:then old:else new:then old:else new:then old:then new:then old:then 100 ✓

28 01 int gt_100(unsigned x) { 02 unsigned y = x; 03 if (change(y > 100, y ≥ 100)) 04 return 1; 05 else 06 return 0; 07 } 28 Shadow Shadow symbolic execution y < 100 y ≤ 100 y < 100 y ≤ 100 y ≥ 100 y ≤ 100 y ≥ 100 y ≤ 100 y ≥ 100 y > 100 y ≥ 100 y > 100 Divergence not always possible y < 100 y > 100 y < 100 y > 100 y > 200

29 shadow  Advantages of shadow symbolic execution  Pruning execution paths – smaller search space  Space efficiency  Two versions combined into one  Expression sharing via shadow expressions  Does not execute unchanged path prefix twice 29 Shadow Shadow symbolic execution

30 Patch annotations

31  Annotations  change(old, new) macro  Currently manual, automation possible  A set of 15 rules  See project web-site for annotated patches http://srg.doc.ic.ac.uk/projects/shadow/ http://srg.doc.ic.ac.uk/projects/shadow/ 31 Expressing patches

32 01 if (change(argc - optind, n_args) < 1) 02 { 03 error (...); 04 usage (EXIT_FAILURE); 05 }  Modifying an rvalue expression 32 Annotation rules Old Combined 01 if (argc – optind < 1) 02 { 03 error (...); 04 usage (EXIT_FAILURE); 05 } New 01 if (n_args < 1) 02 { 03 error (...); 04 usage (EXIT_FAILURE); 05 }

33  Adding an assignment 33 Annotation rules 01 byte_idx = 0; 02 print_delimiter = false; 03 current_rp = rp; OldNew Combined 01 byte_idx = 0; 02 print_delimiter = false; 03 01 byte_idx = 0; 02 print_delimiter = false; 03 current_rp = change(current_rp, rp);

34 Patch testing approach

35 35 Shadow Shadow approach overview Old version New version Test suite

36 36 Shadow Shadow approach overview Unify versions Select test cases Shadow Enhanced checks Old version New version Test suite

37 37 Unify versions Select test cases Shadow Enhanced checks Old version New version Test suite Regression bugs Expected divergences Shadow Shadow approach overview

38 38 Unify versions Select test cases Shadow Enhanced checks Old version New version Test suite Regression bugs Expected divergences Shadow Shadow approach overview

39 39 Unify versions Select test cases Shadow Enhanced checks  Combine old and new version  change() macro  Set of rules Shadow Shadow approach overview

40 40 Unify versions Select test cases Shadow Enhanced checks  Select test cases that touch the patch  Run test suite on the new version  Use coverage data  Cover at least one line of the patch Shadow Shadow approach overview

41 41 Unify versions Select test cases Shadow Enhanced checks  Use test suite inputs Shadow Shadow approach overview

42 42 Shadow Shadow approach overview  Use test suite inputs Unify versions Select test cases Shadow Enhanced checks

43 43  Use test suite inputs  Try to find divergent paths ✓ ✗ ✓ Unify versions Select test cases Shadow Enhanced checks Shadow Shadow approach overview

44 44  Use test suite inputs  Try to find divergent paths  Perform bounded symbolic execution  New test cases  Explore more divergent behaviours ✗ ✓ BSE ✓ Unify versions Select test cases Shadow Enhanced checks Shadow Shadow approach overview

45 45 Unify versions Select test cases Shadow Enhanced checks  Run old and new versions on the generated inputs  Compare:  program outputs  program exit codes  memory safety violations (ASAN) Shadow Shadow approach overview

46 Implementation and evaluation

47  Implemented on top of KLEE  Uses concolic execution functionality from ZESTI and Docovery 47 Implementation http://klee.github.io http://srg.doc.ic.ac.uk/projects/zesti http://srg.doc.ic.ac.uk/projects/docovery

48  Evaluated on patches from CoREBench study  http://www.comp.nus.edu.sg/~release/corebench/ http://www.comp.nus.edu.sg/~release/corebench/  18 unique Coreutils patches which introduced bugs  Significantly more complex than typical patches used in the evaluation of previous work (e.g. SIR, Siemens)  The bug-fixing patches also known  Evaluated 16 out of 18 due to technical issues 48 Evaluation

49 49 Evaluation PatchTool Patch size Annotatio ns LOCHunks 1mv, rm451712 3cut2943514 4tail2144 5=16tail275131 6cut833 7seq14855 8seq37412 10cp1682 11cut211 12=17cut110174 13ls1322 14ls1554 du311 19seq4096 21cut31106 22expr5464

50 50 Evaluation PatchTool Patch size Annotatio ns LOCHunks 1mv, rm451712 3cut2943514 4tail2144 5=16tail275131 6cut833 7seq14855 8seq37412 10cp1682 11cut211 12=17cut110174 13ls1322 14ls1554 du311 19seq4096 21cut31106 22expr5464

51 51 Evaluation PatchTool Patch size Annotatio ns LOCHunks 1mv, rm451712 3cut2943514 4tail2144 5=16tail275131 6cut833 7seq14855 8seq37412 10cp1682 11cut211 12=17cut110174 13ls1322 14ls1554 du311 19seq4096 21cut31106 22expr5464

52 52 Evaluation Generated input Behaviour OldNew cut -s -d: -f0- file contains “:::\n:1” :::\n1\n\n cut –d: -f1,0- file contains “a:b:c” a:b:ca tail --retry ///s\x01\x00g\x00 tail: warning: --retry is useful mainly when following by name… tail: warning: --retry ignored; --retry is useful only when following… Expected

53 53 Evaluation Generated input Behaviour OldNew cut -c1-3,8- --output-d=: file contains “abcdefg” abcabc + buffer overflow cut -c1-7,8- --output-d=: file contains “abcdefg” abcdefgabcdefg + buffer overflow cut -b0-2,2- --output-d=: file contains “abc” abcsignal abort Bugs

54 Generated input Behaviour OldNew cut -c1-3,8- --output-d=: file contains “abcdefg” abcabc + buffer overflow cut -c1-7,8- --output-d=: file contains “abcdefg” abcdefgabcdefg + buffer overflow cut -b0-2,2- --output-d=: file contains “abc” abcsignal abort 54 Evaluation Bugs New bug, not part of CoREBench

55 55 Evaluation PatchDivergences Output differences ExpectedBug 139K3- 315K-- 43936- 5=1614-2 61.4K-86 71245- 854K-- 106-2 118749- 12=174.2K-78 131111 142-- 1511- 1933K7- 2121K151684 22---

56 56 Evaluation PatchDivergences Output differences ExpectedBug 139K3- 315K-- 43936- 5=1614-2 61.4K-86 71245- 854K-- 106-2 118749- 12=174.2K-78 131111 142-- 1511- 1933K7- 2121K151684 22---

57 57 Evaluation PatchDivergences Output differences ExpectedBug 139K3- 315K-- 43936- 5=1614-2 61.4K-86 71245- 854K-- 106-2 118749- 12=174.2K-78 131111 142-- 1511- 1933K7- 2121K151684 22---

58 58 Evaluation PatchDivergences Output differences ExpectedBug 139K3- 315K-- 43936- 5=1614-2 61.4K-86 71245- 854K-- 106-2 118749- 12=174.2K-78 131111 142-- 1511- 1933K7- 2121K151684 22---

59  Unsuccessful cases  Refactorings  Non-functional changes  Memory consumption  Performance  Technical challenges:  Reasoning about file access rights  Symbolic directories support  Floating point support  Not reproducible 59 Evaluation

60 Shadow Shadow symbolic execution  A symbolic execution technique for patch testing  Generates inputs that trigger new behaviours  Prunes large parts of the search space  Useful for: regression testing, test-suite augmentation, patch understanding http://srg.doc.ic.ac.uk/projects/shadow/ Summary

Download ppt "Shadow Shadow of a Doubt: Testing for Divergences Between Software Versions Hristina PalikarevaTomasz KuchtaCristian Cadar ICSE’16, 20 th May 2016 This."

Similar presentations

Shadow Symbolic Execution for Better Testing of Evolving Software Cristian Cadar and Hristina Palikareva Department of Computing Imperial College London.

Shadow Symbolic Execution for Better Testing of Evolving Software Cristian Cadar and Hristina Palikareva Department of Computing Imperial College London.
A Survey of Approaches for Automated Unit Testing

A Survey of Approaches for Automated Unit Testing
A System to Generate Test Data and Symbolically Execute Programs Lori A. Clarke September 1976.

A System to Generate Test Data and Symbolically Execute Programs Lori A. Clarke September 1976.
Abstraction and Modular Reasoning for the Verification of Software Corina Pasareanu NASA Ames Research Center.

Abstraction and Modular Reasoning for the Verification of Software Corina Pasareanu NASA Ames Research Center.
Effectively Prioritizing Tests in Development Environment

Effectively Prioritizing Tests in Development Environment
Using Programmer-Written Compiler Extensions to Catch Security Holes Authors: Ken Ashcraft and Dawson Engler Presented by : Hong Chen CS590F 2/7/2007.

Using Programmer-Written Compiler Extensions to Catch Security Holes Authors: Ken Ashcraft and Dawson Engler Presented by : Hong Chen CS590F 2/7/2007.
(Quickly) Testing the Tester via Path Coverage Alex Groce Oregon State University (formerly NASA/JPL Laboratory for Reliable Software)

(Quickly) Testing the Tester via Path Coverage Alex Groce Oregon State University (formerly NASA/JPL Laboratory for Reliable Software)
ECE 667 - Synthesis & Verification1 ECE 667 Spring 2011 Synthesis and Verification of Digital Systems Verification Introduction.

ECE Synthesis & Verification1 ECE 667 Spring 2011 Synthesis and Verification of Digital Systems Verification Introduction.
Applied Software Project Management Andrew Stellman & Jennifer Greene Applied Software Project Management Applied Software.

Applied Software Project Management Andrew Stellman & Jennifer Greene Applied Software Project Management Applied Software.
Regression testing Tor Stållhane. What is regression testing – 1 Regression testing is testing done to check that a system update does not re- introduce.

Regression testing Tor Stållhane. What is regression testing – 1 Regression testing is testing done to check that a system update does not re- introduce.
Software Testing and QA Theory and Practice (Chapter 4: Control Flow Testing) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.

Software Testing and QA Theory and Practice (Chapter 4: Control Flow Testing) © Naik & Tripathy 1 Software Testing and Quality Assurance Theory and Practice.
CTEC 1863 – Operating Systems Shell Scripting. CTEC1863 - 2009F2 Overview How shell works Command line parameters –Shift command Variables –Including.

CTEC 1863 – Operating Systems Shell Scripting. CTEC F2 Overview How shell works Command line parameters –Shift command Variables –Including.
CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie.

CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie.
Unit Testing & Defensive Programming. F-22 Raptor Fighter.

Unit Testing & Defensive Programming. F-22 Raptor Fighter.
Topics in Software Dynamic White-box Testing Part 2: Data-flow Testing

Topics in Software Dynamic White-box Testing Part 2: Data-flow Testing
Lesson 7-Creating and Changing Directories. Overview Using directories to create order. Managing files in directories. Using pathnames to manage files.

Lesson 7-Creating and Changing Directories. Overview Using directories to create order. Managing files in directories. Using pathnames to manage files.
Testing. Definition From the dictionary- the means by which the presence, quality, or genuineness of anything is determined; a means of trial. For software.

Testing. Definition From the dictionary- the means by which the presence, quality, or genuineness of anything is determined; a means of trial. For software.
CUTE: A Concolic Unit Testing Engine for C Technical Report Koushik SenDarko MarinovGul Agha University of Illinois Urbana-Champaign.

CUTE: A Concolic Unit Testing Engine for C Technical Report Koushik SenDarko MarinovGul Agha University of Illinois Urbana-Champaign.
1 Lab Session-III CSIT-120 Fall 2000 Revising Previous session Data input and output While loop Exercise Limits and Bounds Session III-B (starts on slide.

1 Lab Session-III CSIT-120 Fall 2000 Revising Previous session Data input and output While loop Exercise Limits and Bounds Session III-B (starts on slide.
1 AutoBash: Improving Configuration Management with Operating System Causality Analysis Ya-Yunn Su, Mona Attariyan, and Jason Flinn University of Michigan.

1 AutoBash: Improving Configuration Management with Operating System Causality Analysis Ya-Yunn Su, Mona Attariyan, and Jason Flinn University of Michigan.

Similar presentations

Ads by Google