1. Cristian Cadar, Peter Boonstoppel, Dawson Engler RWset: Attacking Path Explosion in Constraint-Based Test Generation TACAS 2008, Budapest, Hungary ETAPS 2008

2. Goal: generate inputs that explore (ideally) all paths of a program Run program on symbolic input, whose initial value is anything At conditionals that use symbolic inputs, fork execution and follow both paths: On true branch, add constraint that condition is true On false, that it is not When a path terminates, generate a test case by solving the constraints on that path Constraint-Based Test Generation

3. EGT / EXE / KLEE DART [Godefroid/Klarlund/Sen] CUTE [Sen et al.] SAGE, Pex [Godefroid et al.] Vigilante [Castro et al ] BitScope [Song et al.] RWset applicable to any of these Constraint-Based Test Generation

4. Effective bug-finding tool File system code ext2, ext3, JFS Networking applications bpf, udhcpd Library code PCRE, Pintos Device drivers Minix EXE Results

5. Scalability challenge Exponential space! –Relatively small number of interesting paths: e.g., those that achieve maximum branch coverage Mixed symbolic/concrete execution (EXE/DART) Search heuristics –Best First Search (EXE) –Generational Search (SAGE) Symbolic execution + random testing (Hybrid CUTE) Caching function summaries (SMART) Demand-Driven Compositional Symbolic Execution (Pex)

6. RWset (read-write set) analysis Determine whether continuing to execute the current program path will explore new states Only a value observed by the program can determine the execution of new program states

7. {data, arg1, arg2} = unconstrained flag = 0; if (arg1 > 100) flag = 1; if (arg2 > 100) flag = 1; process(data, flag);

8. flag = 0 {data, arg1, arg2} = unconstrained flag = 0; if (arg1 > 100) flag = 1; if (arg2 > 100) flag = 1; process(data, flag);

9. arg1 > 100 flag = 0 {data, arg1, arg2} = unconstrained flag = 0; if (arg1 > 100) flag = 1; if (arg2 > 100) flag = 1; process(data, flag); true: arg1 > 100 false: arg1 100...

10. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 false: arg2 100 {data, arg1, arg2} = unconstrained flag = 0; if (arg1 > 100) flag = 1; if (arg2 > 100) flag = 1; process(data, flag); true: arg1 > 100 false: arg1 100

11. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg2 100 process(data, 1) {data, arg1, arg2} = unconstrained flag = 0; if (arg1 > 100) flag = 1; if (arg2 > 100) flag = 1; process(data, flag); true: arg1 > 100 false: arg1 100

12. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg2 100 process(data, 1) {data, arg1, arg2} = unconstrained flag = 0; if (arg1 > 100) flag = 1; if (arg2 > 100) flag = 1; process(data, flag); true: arg1 > 100 false: arg1 100

13. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg2 100 process(data, 1) If arg1, arg2 not read true: arg1 > 100 false: arg1 100

14. Write-set analysis Look at values written in the past State abstraction in model checking Memory state = write set up to current progr. pt. –Concrete writes: concr loc = concr val –Symbolic writes: constraint(sym loc) Program point P, two paths w/ same write-set –prune the second one

15. Write-sets Precision: reason at the byte-level Minimize write-set size 1. Overwrites 2. Dead locations 3. Alpha renaming Complicated in the symbolic domain a[i] = 17; a[j] = 20; Cannot discard all constraints on dead locations

16. Read-set analysis Key idea: can ignore from write-set writes to locations that are never read again –Definition of read driven by goal to achieve high branch coverage –Location read if can hit a new branch by changing its value

17. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) RWset arg1 = True-True path arg2 = data =

18. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) arg1 = True-True path arg2 = flag = 0 data = RWset

19. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) arg1 > 100 True-True path arg2 = flag = 0 data = RWset

20. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) arg1 > 100 True-True path arg2 = flag = 1 data = RWset

21. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) arg1 > 100 True-True path arg2 > 100 flag = 1 data = RWset

22. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) arg1 > 100 True-True path arg2 > 100 flag = 1 data = RWset

23. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) arg1 > 100 True-True path arg2 > 100 flag = 1 data = RWset

24. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) True-True path flag = 1 data = arg1, arg2 not read arg1 > 100 arg2 > 100 RWset

25. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) True-False path arg1 > 100 True-True path arg2 > 100 flag = 1 data = arg1 > 100 arg2 100 flag = 1 data = arg1, arg2 not read RWset

26. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) True-False path arg1 > 100 True-True path arg2 > 100 flag = 1 data = arg1 > 100 arg2 100 flag = 1 data = arg1, arg2 not read RWset

27. arg1 > 100... arg2 > 100 flag = 1 flag = 0 true: arg2 > 100 flag = 1 false: arg1 100 process(data, 1) true: arg1 > 100 false: arg1 100 process(data, 1) True-False path arg1 > 100 True-True path arg2 > 100 flag = 1 data = arg1 > 100 arg2 100 flag = 1 data = arg1, arg2 not read RWset

28. Implementation Execution path Global cache P W1W1 W2W2 WpWp (W 1, R 1 ) (W 2, R 2 ) … (W n, R n ) RW-Set(P) WPWP

29. Implementation P W1W1 W2W2 WpWp (W 1, R 1 ) (W 2, R 2 ) … (W n, R n ) Write-set Hit! Emit test case i.(W P = W i ) RW-Set(P) WPWP Execution path Global cache

30. Implementation P W1W1 W2W2 WpWp WPWP Read-set Hit! i.(W P R i = W i R i ) Add (W P,R i ) to RW-Set(P) (W 1, R 1 ) (W 2, R 2 ) … (W n, R n ) (W P, R i ) Emit test case RW-Set(P) Execution path Global cache

31. Implementation P W1W1 W2W2 WpWp (W 1, R 1 ) (W 2, R 2 ) … (W n, R n ) (W P, _) Miss! Add (W P,_) to RW-Set(P)... RW-Set(P) WPWP Execution path Global cache

32. Evaluation Medium-sized open source benchmarks –bpf, udhcpd, expat, tcpdump, pcre Minix 3 device drivers –lance, pci, sb16

33. Medium-sized apps bpf: Berkeley Packet Filter expat: XML parsing library pcre: Perl compatible reg exp library tcpdump: tool for printing packet headers udhcpd: a DHCPD server

34. Medium-sized apps Ran base EXE on each app for 30 minutes –Except 30,000 test cases for PCRE Recorded number of branches hit Reran in RWset mode until we reached the same branch coverage

35. Medium-sized apps

36. Non-redundant states pcre tcpdump expat udhcpd bpf Test cases Non-redundant states Test cases Non-redundant states

37. Performance Dry run for the medium-sized apps: compute write-sets and read-sets but do no pruning

38. Device drivers Hard to check w/o the physical device or outside of the kernel Focused on three MINIX 3 drivers: –lance: AMD lance ethernet card driver –pci: PCI bus driver –sb16: Sound Blaster 16 driver

39. Minix 3 device drivers Structured around a main dispatch loop while (1) { /* receive message from other processes, the kernel, or the hardware */ message = read_message(); process(message); }

40. Minix 3 device drivers Structured around a main dispatch loop while (1) { /* receive message from other processes, the kernel, or the hardware */ message = read_symbolic_data(); process(message); }

41. Minix 3 device drivers Structured around a main dispatch loop for (k=0; k < n; k++) { /* receive message from other processes, the kernel, or the hardware */ message = read_symbolic_data(); process(message); }

42. Minix drivers - evaluation Three versions of EXE: –Base –Write-set –RWset Fixed the # of iterations in each version –mainly to have the base version terminate Ran each version for an hour –recorded branch coverage + non-redundant states

43. Branch coverage (pci) RWset Write-set Base

44. Branch coverage (lance) RWset Write-set Base

45. RWset Write-set Base Branch coverage (sb16)

46. Non-redundant states RWset DFS RWset DFS RWset DFS RWset Base RWset Base RWset Base

47. Bugs in device drivers

48. Summary RWset analysis –efficient way to prune large numbers of redundant paths –only values observed by the program trigger the execution of new paths Evaluated on Minix drivers and medium size applications –big reduction in the number of paths explored

49. Questions?