1. Experimentation and Operational Research
OR Society Defence Study Group
10 June 2004
George Pickburn
Information Management Department

2. Topics Operational Research / Analysis - what is it ? Experimentation
Principles - why experiment ?
‘Applications’ and Experiment ‘types’
Facilities
Experiments in the programme
Experimentation as a research Theme
The UK Code of Best Practice
24 November November 2018

3. “There is no empirical method without speculative concepts and systems;
and there is no speculative thinking whose concepts do not reveal, on closer investigation, the empirical material from which they stem.”
Einstein (1953), in his introduction to an edition of Galileo’s Dialogue Concerning the Two Chief World Systems (1632)
24 November November 2018

4. Operational Research / Analysis
CSA’s Guidelines :
“OA contributes to the basis on which MOD makes its decisions by providing an open, explicit and objective assessment of the military capability provided by a proposed course of action “
“in combination with cost data quantitative comparison . . (of) . . new capabilities”
“OA is based . . on objective data about equipment, its human operators and their operational environment”
“The idea of the model lies at the heart of OA a simplified representation of reality . . to allow us to gain understanding of the factors relevant to (a) decision”
A scientific process of analysis supporting decision making
24 November November 2018

5. What’s new? New . . . . . How do people use information ?
op concepts
information facilities and services
ways of working - joint and combined
doctrine
How do people use information ?
Which information do they use ?
24 November November 2018

6. Analysis Techniques Spectrum
Experiments
Experiments
War
Synthetic Environments
Simulation models
Exercises
Mathematical Models
Wargames
High
Realism
Cost
High
Low
Low
Low
High
Repeatability
24 November November 2018

7. What is experimentation?
Controlled process of interaction with and observation of the real world
Twin modes of enquiry / experimentation : inductive, and hypothetico–deductive
Complementary nature of these modes –
most new knowledge arises as (provisional) insight derived from ‘casual’ observation
insights formally validated and tested through experimentation
any one “experiment” can do both
24 November November 2018

8. Principles - why experiment ?
Standard part of the general purpose research enterprise
Generation of new knowledge
Consolidation of knowledge in revised and validated theories and models
This is a recursive, not a linear process : model-test-model-test
24 November November 2018

9. Model - Test - Model Paradigm
Knowledge
Requirements
New
Real World
Experiments
Analytical
Models
Environment of executive decision making,
studies, concept definition, op. planning,
procurement
(Analytical)
Models/Theories
(OA) Studies
Experimentation
24 November November 2018

10. “Couldn’t interface with a new paradigm,
whatever the hell that means “
24 November November 2018

11. ‘Applications’ - Experiment ‘types’
exploration / discovery / visualisation
test & evaluation,
formal verification and validation,
demonstration
systems design
warfighting experiments - VCDS policy
all are experiments to meet differing requirements for confidence / burden of proof
24 November November 2018

12. Experimentation typology
Burden of proof
Quantity of knowledge
Quality of knowledge
Confidence
# 3
Experimentation Campaign
Expt
# 2
Tests
Trials
Exercises
Demos
# 1
No. of good ideas
Visualisations
Discovery
No. of hypotheses
24 November November 2018

13. CCRP CoBP Experimentation
24 November November 2018

14. What changes between types? ( = design )
Quantity (categories) of knowledge gained
Quality (confidence) of knowledge gained (‘ burden ‘ of proof)
Both are controllable to meet the purpose of the experiment - how will the knowledge gained be used?
There is always a hypothesis
Design entails trade-off of:
effect size,
confidence (in the truth of the research hypothesis),
power to reject false hypotheses, and
sample size
24 November November 2018

15. Facilities Dstl Applied Research Technology Demonstrator (ARTD)
C2 systems lab
NITEworks
an experiment on three levels
Exercises as an experimentation environment of opportunity - BATUS & Warfighting experiments
Wargames / Training Systems
Combined Arms Tactical Trainer
24 November November 2018

16. Experiments in the programme
Kill Chain - Responsive C2 of tactical air under different sensor ‘treatments’
Indirect Fire Integration - new munitions, role of intelligence, targeting facilities
Future HQ - organisation and design
Effects Based Planning - MNE3
ISTAR tasking and integration
24 November November 2018

17. Experimentation as an R&D theme
Coherence for efficient R&D
Exploitation of exercises
Standards across the programme
Output sharing and re-use
Theme Adviser on experimentation
24 November November 2018

18. Codes of Best Practice NATO CoBP for C2 Modelling
US Code of Best Practice
US Marine Corps “Practice of Military Experimentation”
Facility CoBPs - UK ARTD, US JFCOM
TTCP CoBP for Experimentation - soon
UK MoD Code of Best Practice
24 November November 2018

19. Experimentation Best Practice I
Identify the knowledge being sought
what is weak / wrong in our current models
use model to provide an expectation to be tested by an experiment
Identify the relevant variables
Form hypotheses (testable ideas) about these variables
Design experiments to test the hypotheses - in the light of the expected uncontrollable variability - using statistical methods
Capture the design and its rationale in a design document for peer review
24 November November 2018

20. Experimentation Best Practice II
Record the conclusions about the tested hypotheses (for later re-use)
Amend the encapsulating theory or model in the light of the proven hypotheses
= the model-test-model paradigm
24 November November 2018

21. Summary Experimentation is . . . . .
the best opportunity for a generation to explore interactions of people and information to enable
balanced exploration of the lines of development
better military organisation design
better procurement of information and communication systems
effective military and government response to emergent ‘asymmetric’ threats
extension and validation of the models for decision making
the natural (and essential) complement of operational research / analysis
24 November November 2018