1. The Evidence Framework Approach: Practical Ways of Thinking About Evidence th International Symposium on Military Operational Research 18th to 21st July 2017 Royal Holloway, University of London
Paul Pearce
Simulation & Numerical Methods Group
Defence and Security Analysis Division
Defence Science & Technology Laboratory
Portsdown West
19 September 2018
© Crown copyright 2017 Dstl
DSTL/PUB103348

2. Operational Analysis (OA) Focus
Uncontrolled copy when printed
19/09/2018
Operational Analysis (OA) Focus
Dstl’s Defence & Security Analysis Division applies OA to a range of questions using a variety of ‘soft’ and ‘hard’ analysis techniques….presenting a number of evidence challenges.
Investment requires evidence
2017
2037+
19 September 2018
© Crown copyright 2017 Dstl
© Crown copyright 2014 Dstl

3. The Evidence Framework Approach
Problem Formulation
Become better system thinkers through enhanced Study design approach
Quality of Evidence
Employ a means to characterise quality of evidence using the Evidence Profile Table (EPT), Validation Profile Table (VPT) and Confidence Assessment Table (CAT)
Quality Assurance
Think about analytical quality assurance by assessing RIGOUR* using an Evidence Quality Questionnaire (EQQ)
Results in better engagement, greater understanding and more robustly defendable evidence
* Repeatability Independence Grounding in reality Objectivity Uncertainty-managed Robustness
19 September 2018
© Crown copyright 2017 Dstl

4. Problem Formulation (Enhancing Study Design).
Through conducting an Analysis Estimate
David Snowden -
19 September 2018
© Crown copyright 2017 Dstl

5. Uncontrolled copy when printed
19/09/2018
Quality of Evidence - Using the EPT
Assessment of an analytical programme of work using the Evidence Profile Table
Problem Space Obvious
All uncertainty explored
System can be described
Problem Space chaotic
Unknowable-unknowns perspective
Increasing certainty
19 September 2018
© Crown copyright 2017 Dstl
© Crown copyright 2013 Dstl

6. Uncontrolled copy when printed
19/09/2018
Target 10
Quality of Evidence - Using the EPT
Assessment of an analytical programme of work using the Evidence Profile Table
Actual 11
Comprehensiveness
Relevance
Objectivity
Quantity
Consistency
19 September 2018
© Crown copyright 2017 Dstl
© Crown copyright 2013 Dstl

7. Uncontrolled copy when printed
19/09/2018
Target 6
Quality of Evidence - Using the VPT
Actual 7
Assessment of an analytical programme of work using the Evidence Profile Table
Construct Validity
Content Validity
Face Validity
Criterion Validity
19 September 2018
© Crown copyright 2017 Dstl
© Crown copyright 2013 Dstl

8. Uncontrolled copy when printed
19/09/2018
Quality of Evidence – Using the CAT
Assessment of an analytical programme of work using the Confidence Assessment Table
Moderate warrant
Strong Validity
Medium Confidence
19 September 2018
© Crown copyright 2017 Dstl
© Crown copyright 2013 Dstl

9. Example Application – LEOA Insights EPT only
Method
Coverage of relevant factors
Good track record for problem.
Strong direct support and indirect support for insights.
Simulation mode: Possible to assert that A is very likely to cause B for some of the insights.
Insights
Insight 1 has arisen in other studies and in other scenarios. Further analysis unlikely to change the findings.
Insight 2 has emerged over several LEOA activities.
Insight 3 new or emerging insight not been seen before.
Study
Target level of evidence required
Multiple experiments – the production of capping paper to consolidate findings
Draw on further expertise to validate assumptions before use and after findings
19 September 2018
© Crown copyright 2017 Dstl

10. Quality Assurance Uses the Evidence Quality Questionnaire (EQQ).
The EQQ is a comprehensive framework for assessing analytical quality against key criteria, helping users to understand how the evidence should be interpreted and to identify areas for improvement.
Structured around the mnemonic RIGOUR as used in the Aqua Book.
EQQ has sections corresponding to the generic phases of a study.
19 September 2018
© Crown copyright 2017 Dstl

11. EFA Summary
The Evidence Framework Approach will improve the quality of OA to inform decision making
Extensive research has been conducted to provide the conceptual underpinnings
Its utility has being tested through experimentation and application
Guidance is in place to aid its use on studies
Development continues and your help in taking it forward and improving it is welcome
19 September 2018
© Crown copyright 2017 Dstl

12. What Next?
The EFA is being successfully applied and is continuing to develop:
The EFA is transitioning to ‘business as usual’ influencing a number of communities of interest.
Engagement with academia, industry and analytical communities continues to test and develop the EFA, improving the language and broadening the applicability.
Engagement with wider UK Government:
Aiming to provide input for updates to the Governments guidance on providing quality analysis (AQUA Book)
ISMOR Workshop tomorrow (19th) at 0900 and 1330 to introduce the ideas
19 September 2018
© Crown copyright 2017 Dstl

13. Further Reading
19 September 2018
© Crown copyright 2017 Dstl