1. Teaching with angr: A Symbolic Execution Curriculum and CTF
Jacob M. Springer1,2, Wu-chang Feng1
1Portland State University
2Swarthmore College
Teaching with angr
we made a series of capture the flag levels and an associated curriculum designed to teach symbolic execution techniques to students
This CTF isn’t like a typical CTF
It’s pedagogical, not evaluative
It’s scaffolded and metamorphic—we will discuss these

2. What is symbolic execution?
Outline
What is symbolic execution?
How do we teach it?
I want to talk about two things in this presentation
First, symbolic execution
Next, now do we teach it?

3. Symbolic execution: why should you care?
First, what is the point of symbolic execution?
Technique to search programs to discover vulnerabilities/bugs automatically
Check for correctness
Important in security and program verification

4. Symbolic execution: why should you care?
Program analysis and testing

5. Symbolic execution: why should you care?
Program analysis and testing
Microsoft applications (PowerPoint, Word, etc.)

6. Symbolic execution: why should you care?
Program analysis and testing
Microsoft applications (PowerPoint, Word, etc.)
DARPA's Cyber-Grand Challenge

7. Symbolic execution: why should you care?
Program analysis and testing
Microsoft applications (PowerPoint, Word, etc.)
DARPA's Cyber-Grand Challenge
Important for students to understand and apply

8. What is symbolic execution?

9. Find input to print “Good Job.”
A simple CTF level source code
Goal is to print “Good Job.”
Students receive this as a binary
Must reverse engineer the check_code function to return 1
Symbolic execution is CTFs for the lazy
Tell it you are looking for when the program prints “Good Job”
It searches it for you and gives you a solution

10. Execution paths can be represented as a tree.
if (input > SECRET+100)
if (input == SECRET+68)
if (input < SECRET)
if (input <= SECRET+78)
if (input & 0x1)
if (input & 0x2)
if (input & 0x4)
return 0;
return 1;
if (input >= SECRET+88)
Execution paths can be represented as a tree.
#define SECRET 100
int check_code(int input) {
    if (input >= SECRET+88) return 0;
    if (input > SECRET+100) return 0;
    if (input == SECRET+68) return 0;
    if (input < SECRET) return 0;
    if (input <= SECRET+78) return 0;
    if (input & 0x1) return 0;
    if (input & 0x2) return 0;
    if (input & 0x4) return 0;
    return 1; }
How does it do it?
Can represent a program as a directed graph
Children of nodes are the next possible instructions
A node has two or more children when it branches
Thus, finding a solution is a search problem

11. Animation: Building a Set of Paths
Legend:
Blue = already executed
Yellow = active
Red = terminated
if (input >= SECRET+88)
Start with the first instruction

12. Animation: Building a Set of Paths
Legend:
Blue = already executed
Yellow = active
Red = terminated
if (input >= SECRET+88)
return 0;
if (input > SECRET+100)
Since we branch, we look at the two possibilities

13. Animation: Building a Set of Paths
Legend:
Blue = already executed
Yellow = active
Red = terminated
if (input >= SECRET+88)
return 0;
if (input > SECRET+100)
return 0;
if (input == SECRET+68)
When we find a “return 0” (since we know we don’t want it) or reach a leaf of the tree, we stop looking at it

14. Animation: Building a Set of Paths
Legend:
Blue = already executed
Yellow = active
Red = terminated
if (input >= SECRET+88)
return 0;
if (input > SECRET+100)
return 0;
if (input == SECRET+68)
return 0;
if (input < SECRET)
We found what we wanted!
return 0;
if (input <= SECRET+78)
return 0;
if (input & 0x1)
return 0;
if (input & 0x2)
return 0;
if (input & 0x4)
Eventually, we find what we want
return 0;
return 1;

15. Applying symbolic execution
Once we have a path, we can build an equation that can be solved by the computer:
if (input >= SECRET+88)
if (input > SECRET+100)
if (input == SECRET+68)
if (input < SECRET)
if (input <= SECRET+78)
if (input & 0x1)
if (input & 0x2)
if (input & 0x4)
return 1;
input >= SECRET+88
∧ input > SECRET+100
∧ input == SECRET+68
∧ input < SECRET
∧ input <= SECRET+78
∧ input & 0x1
∧ input & 0x2
∧ input & 0x4
Left: the path we found
Right: an equation built from the path
Solving the equation yields an input that will lead the program to print Good Job

16. Angr-y CTF
Goal: Build a curriculum and a set of capture-the-flag (CTF) levels to introduce students to symbolic execution

17. Our Approach

18. Our Approach Modeled after MetaCTF (USENIX 3GSE 2015)
Find a password that causes a program to print "Good Job."
Modeled after MetaCTF
Built by Prof. Wu-chang Feng at Portland State University
Levels are scaffolded and metamorphic (we will talk about this soon, I promise)
All levels require student to find input that prints Good Job

19. Our Approach Modeled after MetaCTF (USENIX 3GSE 2015)
Find a password that causes a program to print "Good Job."
18 scaffolded levels
Requires symbolic execution to solve
We organize it into 18 scaffolded levels

20. Our Approach Modeled after MetaCTF (USENIX 3GSE 2015)
Find a password that causes a program to print "Good Job."
18 scaffolded levels
Requires symbolic execution to solve
Uses angr (angr.io)
We use the angr symbolic execution engine, though we focus on concepts rather than implementation within angr
So, these levels should generalize to any symbolic execution tool

21. A typical level
What does a typical level look like?

22. A typical level
Student receives a binary and a template angr script

23. A typical level Student receives a binary and a template angr script
Student edits the template to analyze the binary

24. A typical level Student receives a binary and a template angr script
Student edits the template to analyze the binary
Student runs the script which prints a password

25. A typical level Student receives a binary and a template angr script
Student edits the template to analyze the binary
Student runs the script which prints a password
Student runs the binary and types in the password to confirm their work

26. The levels are scaffolded

27. What does scaffolding mean?
Support structure, just like a scaffold
Guided, incremental introduction of concepts
Scaffolding, much like a real-life scaffold, provides structure for students
Guides students through challenges
We will look at examples
On the right, it is a screenshot of the level names from the GitHub repository
Even in their name, they give hints about how to solve them

28. CTF Modules Basic symbolic execution Symbol injection
Handling complexity
Automated exploitation
Organized into four categories

29. Scaffolding for pedagogy: not frustrating
Level 1
Well documented
Only need to change two lines
This is the first level
There are a lot of lines of code
HOWEVER
Mostly comments documenting each line
Student only needs to change two lines (in red)
Description tells the student how to solve the level, requires that student understand that description
No emphasis on syntax or implementation details

30. Scaffolding for pedagogy: guided
Tells student how to get started
This is an example of a conceptual description of a level
Take a moment to read this
We want to find an arbitrary write vulnerability
Challenges are not open-ended (and therefore not frustrating)
Student should already know how to do this manually
Taught previously in the class

31. Scaffolding: simple This is the first level
Important part of the code are last two lines
Student needs to find the instruction address of where the program prints “Good Job”
Then, angr will search for it using .explore

32. MetaCTF Example
Remember this example? It compiles into the assembly on the right
Student can find this address and plug it into the script

33. Scaffolding: builds on previous concepts
Plugging it into the script is shown here
Very simple first level, helps build confidence

34. Scaffolding: incremental and reinforcing
Level 02 (find_condition)
Level 03 (symbolic_registers)
I will show you the strategy for solving two sequential levels
The point is to demonstrate that we add on concepts incrementally and then use reinforcement by repetition

35. Scaffolding: incremental and reinforcing
Level 02 (find_condition)
1. Load binary
2. Define the termination condition (Has the program printed “Good Job.”?)
3. Search binary for condition
Level 03 (symbolic_registers)
Read slide

36. Scaffolding: incremental and reinforcing
Level 02 (find_condition)
1. Load binary
2. Define the termination condition (Has the program printed “Good Job.”?)
3. Search binary for condition
Level 03 (symbolic_registers)
2. Inject symbols
3. Define the termination condition (Has the program printed “Good Job.”?)
4. Search binary for condition
As you can see, level 03 has the same solution as level 02,
but requires student to inject a symbol,
introduces that concept and reinforces the previous one

37. Scaffolding: conceptual
First glance: seems complicated
At first glance, this seems complicated
Goal of this level is to find a place where we can make an arbitrary instruction pointer jump

38. Scaffolding: conceptual, part 2
This still might look complicated, but really, it’s simple
First, get the instruction pointer reference
Check if we can find an input that will cause the instruction pointer to jump to where we want
Constrain the instruction pointer to be our desired value
(not shown) solve for the input

39. What does metamorphic mean?
Different SECRET for every student
Can generate arbitrary C code
Metamorphic is a fancy word for meaning that each binary has a different password
Different SECRET for every student
Notice that the difficulty is consistent
We can also generate arbitrary C code (and we do!)

40. Metamorphic levels Reduce cheating Allow reuse
Maintain consistency of difficulty across students

41. Evaluation
Offered Winter 2018 in Portland State University's CS 492/592: Malware course
Last 2 weeks focused on symbolic execution
Survey given at the end of two weeks
33 of 42 responded

42. Results
Curriculum and scaffolding allow students to complete most levels
Completion percentage
Number of students
95-100% 25
85-95% 4
75-85% 6
Below 75% 7

43. Survey Ratings evaluate helpfulness of curriculum and CTF
Very Unhelpful = 1
Very Helpful = 5
Q1: Rate the lecture material for understanding the concepts
Q2: Rate the CTF exercises for understanding the concepts
Rating 1 2 3 4 5
Mean Q1 17 12
4.15
Rating 1 2 3 4 5
Mean Q2 18 9
3.94

44. Survey
Q3: Rate the CTF exercises for developing skills in using symbolic execution techniques
Rating 1 2 3 4 5
Mean Q3 16 10
3.94

45. https://malware.oregonctf.org Try the CTF!
Also on GitHub