1. test oracles Andrew J. Ko Brian P. Robinson Jamie Andrews Mark Grechanik Michal Young

2. 2 what’s a test oracle? a decision procedure on a test execution (defined by an I/O pair, predicted outputs, internal states, logs, etc.), that makes a yes/no decision about the execution’s correctness without them, you have no idea what you’re testing for! oracle yes/no

3. 3 where do they come from? oracle developers come to consensus about them either on their own, through consensus with the rest of the team yes/no

4. 4 focus on functional qualities is too narrow many approaches focus on crashes and other easy to test functional properties oracle = yes/no crashes and other easy test functional properties

5. 5 need to expand to other qualities oracle = yes/no confusing illegible no feedback performance issue functional quality unmaintainable insecure boring compliant...

6. 6 need to expand to other sources oracle requirements specifications expert human judgement industrial data sources logs code runtime... = confusing illegible no feedback performance issue functional quality unmaintainable insecure boring compliant... yes/no

7. 7 67% likely maybe yes/no relative judgement need to consider non-binary outcomes oracle = confusing illegible no feedback performance issue functional quality unmaintainable insecure boring compliant... requirements specifications expert human judgement industrial data sources logs code runtime...

8. 8 oracleoracleoracleoracle oracles may change over time consensus may change or requirements may drift; this demands test oracle maintenance and evolution oracle time yes/no

9. 9 how oracles are assessed vary by quality unique measurements unique oracle representations unique use of human judgements research should be explicit about the type of qualities addressed

10. 10 embedding semantic data in oracles about database value semantics how do you encode the semantics to find bugs?

11. 11 extracting and improving oracles from expert judgements pick a domain! database constraints satellite photos spreadshet outputs medical images user interfaces might be difficult to do because of small numbers of expert judges

12. 12 extracting non-functional oracles from existing systems approach depends on quality performance might look at use cases usability qualities might be discerned from user interface use cases

13. 13 extracting performance oracles for intermediate units machine learning on path characteristics

14. 14 assessing test suite representativeness with feature usage data identifying missing tests or inadequate coverage of high use features

15. 15 evolving maintainable oracles determine tests and test oracles that balance correctness with brittleness again, may depend on quality being tested e.g., brittleness one thing in user interfaces and something else in performance

16. 16 UI feedback oracles test cases that ensure that for every user input, there is a corresponding system output along all control flow paths

17. 17 proposals extracting unit-level performance oracles configuration-aware oracles extracting usability test oracles operationalizing several key usability measurements for oracles extracting oracles from human judgements in... medical imaging, financial analytics, GIS, ?