1. Dror Feitelson The Hebrew University of Jerusalem
Eye Tracking
and
Program Comprehension
Dror Feitelson
The Hebrew University of Jerusalem

2. The ratio of time spent reading vs. writing is well over 10:1
The ratio of time spent reading vs. writing is well over 10:1. We are constantly reading old code as part of the effort to write new code.
Robert C. Martin
(Uncle Bob)
Clean Code
Photo: Tim-bezhashvyly

3. Consequences:
Program comprehension is important
Program comprehension is a major factor in software development costs
But…
Program comprehension is hard

4. Why
software is
hard to ?
understand

5. On the Menu: Experiments Definitions Results Methodology Ideas Code
comprehension
Eye
tracking

6. Code Complexity The property of code that makes it hard to comprehend
Using goto, pointers, recursion
Code smells
Just having a lot of it
Identified and measured by complexity metrics
Static analysis
Overly complex code = warning that code is “bad”
= danger of bugs

7. McCabe Cyclomatic Complexity (MCC)
Given a strongly connected
directed graph G with
n nodes, e edges, p connected components
the cyclomatic number of the graph is
v(G) = e – n + p
Calculate this on a module’s control flow graph
Shortcut: in planar (structured) graph count branch points + 1
Suggestion: keep MCC  10
Strongly connected: can get from any node to any other node, in code need to connect exit node back to entry node

8. BUT…
It is reasonable to give the same weight to all branching constructs?
while loop
for loop
if-then-else
case in switch
Is control flow the only thing that matters?
It’s so 1970s…

9. We Can Do Better Measure the effect of different code elements
Express the same functionality in different ways
Conduct controlled experiments
Compare the results
Time to comprehend
Probability to make an error
S. Ajami, Y. Woodbridge, and D. G. Feitelson, “Syntax, predicates, idioms --- what really affects code complexity?”. Empirical Softw. Eng. 24(1), pp , Feb 2019

10. Example 1: Loop Idioms Variations on for (i = 0; i < n; i++)
Loops counting down take more time
Abnormal versions cause more errors

11. Code complexity depends on expectations!
Example 1: Loop Idioms
Variations on for (i = 0; i < n; i++)
Loops counting down take more time
Abnormal versions cause more errors
Code complexity depends on expectations!

12. Complexity and Expectations
“Code complexity depends on expectations”
Is this really a form of “complexity”?
Actually, misleading code causes more errors
Comprehension time as metric for complexity
Error rate as metric for complexity

13. Example 2: Code Regularity
Regularity = the same structures repeated time after time

14. Example 2: Code Regularity
Regularity = the same structures repeated time after time
Hypothesis: Figuring out the basic repeated element makes it easier to understand additional instances

15. Example 2: Code Regularity
Code complexity depends on context!
Regularity = the same structures repeated time after time
Hypothesis: Figuring out the basic repeated element makes it easier to understand additional instances

16. Example 2: Code Regularity
Same style of controlled experiment
Additional metric: gaze pattern
Distribution of fixations
Total time of fixations
Measure time devoted to repeated elements
A. Jbara and D. G. Feitelson, “How programmers read regular code: A controlled experiment using eye tracking”. Empirical Softw. Eng. 22(3), pp , Jun 2017

17. Regular version Non-regular version
Image processing code: replace each pixel with median of 3x3 neighbors; avg of 4-5 subjects

18. in each additional instance
Exponential model:
Time reduced by 40%
in each additional instance

19. Results So Far We can perform detailed measurements
Empirically quantify code complexity
Code complexity ≠ code “misleadingness”
Prefer time over errors as metric
We can use eye tracking data to parameterize models
e.g. “effective complexity” based on instance number

20. However…
The model that comprehension depends on code complexity is too simplistic
Comprehension more realistically depends on the combination of 3 factors:
The code’s complexity
The developer’s skill and domain knowledge
The circumstances (fatigue, priming)

21. Easy Hard Code complexity Developer skill Transient conditions
Technical property of code
Code complexity
Easy
Developer skill
Properties of HUMANS
Hard
Transient conditions
Eye tracking can help
But need careful consideration of code and task

22. code
code

23. Single Function Code Used in most code comprehensions studies
Used in most eye tracking studies
Fits on one page / screen
Somewhat artificial situation
When do you consider a function in isolation?

24. Eye Tracking Results So Far
Non-linear order of reading
Unlike mostly linear order for normal text
Use of recurring patterns
Linear scan of body of code (general overview)
Jump back (to declarations?)
Jump forward (to return value?)
Hover (focus on a certain statement)
Huge differences between subjects
Single function – replicated results

25. EMIP 2013 Data
Two subjects given different tasks

26. EMIP 2013 Data
1st subject

27. Different people reading the same code

28. Full System What does it mean to comprehend a system?
More about structure than statements
The components comprising the system
Their interactions
Data flow
Largely detached from the code itself
Related to the system’s architecture
Eye tracking not very relevant
O. Levy and D. G. Feitelson, “Understanding large-scale software -- A hierarchical view”. In 27th Intl. Conf. Program Comprehension, May 2019

29. In Between More realistic developer workflow Using an IDE
Functions in the context of classes and packages
Follow control flow from one function to another
Using an IDE
Sequence of jumps between modules
What is read in each one
Also other tools (issue tracking, Google)
Eye tracking tools just beginning to emerge
iTrace by Maletic and Sharif
Enabler of new data collection

30. task
task

31. What Does “Reading” Mean?
How do you read
A WhatsApp conversation
A novel
A newspaper story
A poem
A scientific paper
Code
Do you read every word, or just skim most of it?
Do you fully understand it?

32. Levels of Understanding
How do I use this function?
Black box / information hiding
Very common (libraries etc.)
How does this function work?
White box
Superficial: given arguments, what does it print?
Deep: how does it compute its output?
Interpretation
Comprehension

33. Interpretation vs. Comprehension
Interpretation: follow control flow and interpret commands in sequence
Technical process
Comprehension: articulate the essence of what the code does
Deep understanding

34. Comprehension can be easier than interpretation!
int sum=0; for (int i=1; i<=100; i++) { sum += i; } print( sum );
Comprehension can be easier than interpretation!

35. Performing a task is proof of a certain level of comprehension
You need different levels of comprehension for different tasks

36. Answer a trivia question
E.g. does the code include a variable “fooBar”
Need to scan the code
Eye tracking will probably reflect scanning
Does not reflect understanding

37. What does this code print?
Need to trace and interpret the code
Eye tracking will reflect tracing of control flow
Perhaps with shortcuts
(e.g. don’t repeat loops)

38. How do I use this code? Treat code as black box
Depends on API and usage examples
Eye tracking will probably show focus on signature

39. I need to fix a bug in the code
Simple case: just trace the code
e.g. to find a null pointer reference
Hard case: need to really understand it
Same as other hard scenarios

40. Assess quality of the code
As in the context of a code review
Do you need to really understand it?
Can you just scan to search for problem indicators?
Eye tracking may reflect linear scan

41. Modify, fix, or document Need to really understand the code
(white box)
Eye tracking will reflect the effort
Scan the code, jump forward, jump back
Different patterns for different people

42. people
people

43. In addition to the code and the task
how code is read
may depend on
WHO is reading it

44. Gender Do women read code differently from men? Few studies
Some differences
Women perhaps more methodical
e.g. start with problem statement
Men tend to jump ahead

45. Novice vs. Expert
Do expert developers read code differently from novices?
Some studies
Some differences
Experts identify and focus on the important parts
Experts read code less linearly

46. The BIG Question for Eye Tracking:
they
What
are
actually
doing ?

47. Humans Developers’ goal is to “get the job done”
They want to do this efficiently
So may actually try to avoid comprehension, because comprehension is hard
Eye tracking results may reflect activity to circumvent comprehension rather than activity to achieve comprehension

48. EMIP Coding Scheme for Strategies
AttentionToDetail
DataFlow
Debugging
LinearHorizontal and LinearVertical
Deductive
DesignAtOnce
FlowCycle
Inductive
InterproceduralControlFlow
IntraproceduralControlFlow
StrayGlance
TestHypothesis
Touchstone
Trial&Error
Wandering
“It is a capital mistake to theorize before one has data.” –Sherlock Holmes

49. Collecting Lots of Data
Needed for small effects and many confounding factors
Identify common vs. uncommon cases
Analyze effects and interactions
Increase confidence via reproducibility
EMIP dataset example
Is this “Big Data”?

50. Goal: scientific basis for software engineering
Wrapping Up
Goal: scientific basis for software engineering

51. Wrapping Up Subgoal: understand program comprehension
Goal: scientific basis for software engineering
Subgoal: understand program comprehension

52. Wrapping Up Means: controlled experiments
Goal: scientific basis for software engineering
Subgoal: understand program comprehension
Means: controlled experiments

53. Wrapping Up Prerequisites: Define tasks, metrics, variables
Goal: scientific basis for software engineering
Subgoal: understand program comprehension
Means: controlled experiments
Prerequisites:
Define tasks, metrics, variables

54. Wrapping Up Tools: eye tracking Prerequisites:
Goal: scientific basis for software engineering
Subgoal: understand program comprehension
Means: controlled experiments
Prerequisites:
Define tasks, metrics, variables
Tools: eye tracking

55. Wrapping Up Observation: we know so little Tools: eye tracking
Goal: scientific basis for software engineering
Subgoal: understand program comprehension
Means: controlled experiments
Prerequisites:
Define tasks, metrics, variables
Tools: eye tracking
Observation: we know so little

56. Wrapping Up consequence: there’s so much to do!
Goal: scientific basis for software engineering
Subgoal: understand program comprehension
Means: controlled experiments
Prerequisites:
Define tasks, metrics, variables
Tools: eye tracking
Observation: we know so little
consequence: there’s so much to do!