1. SponsorProblem AssessRisk SolutionStrategy Measures of Merit (MoM) Human & OrganisationalIssues Scenarios Methods & Tools Data Products 3 6 4 5 7 10 8 9 11 ProblemFormulation Prepare for Success Data Chapter 9 Stuart Starr MITRE NATO Code of Best Practice (COBP) for C2 Assessment

2. C O B P 2 Data Sponsor Problem Assess Risk Problem Formulation Solution Strategy Measures of Merit ( MoM ) ) Human & Organizational Issues Scenarios Models Methods & Tools Data Products 3 6 4 5 7 10 8 9 11 Prepare for Success

3. C O B P 3 Agenda Definitions, Taxonomy Nature of the Problem Key Needs, Best Practices Summary

4. C O B P 4 Definitions “Data are factual information that are organized for analysis and in a form suitable for machine processing” “Metadata are information about information; e.g., –Documentation of the attributes of data directly attached to the data –Characterization about reliability of the data”

5. C O B P 5 Data Taxonomy AreaExamples Sources Official Legacy Study Results Open Subject Matter Experts (SMEs) Data Types Raw Processed Aggregate Statistical Derived Intermediate Assessment Data Domains Scenario Human/Organizational C2 Performance Tool Measures of MeritDP - MOP - MOCE - MOFE - MOPE

6. C O B P 6 Nature of the Data Problem (1 of 2) “The government is very keen on amassing statistics. They collect them, add them, raise them to the n-th power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman, who just puts down what he damn pleases.” -- Comment of an English judge on the subject of Indian statistics; Quoted in Sir Josiah Stamp in “Some Economic Matters in Modern Life”

7. C O B P 7 Nature of the Data Problem (2 of 2) “Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.” Sherlock Holmes “Metadata! Metadata! Metadata!” Simone Youngblood, MORS Workshop “Theory without data = philosophy; data without theory = noise” Anonymous

8. C O B P 8 Selected Data Problems The data problems that the C2 assessment community are facing are not unique to it -- many other communities seem to have the same issues, including, inter alia, –Data acquisition –Data conversion –Data sharing –Data reuse –Data purity –Metadata policy –Data shelf life –Data naming conventions –Data reconciliation – Data maintenance – Data protection – Data provenance – Data surrogation – Data bloat – Lack of knowledge of original purpose – Lack of good data dictionaries – Ontological development for intelligent searches

9. C O B P 9 Key Data Issue: Barriers to Reuse Lack of knowledge about existence of legacy data Security restrictions Quality of metadata (e.g., failure to document conditions of collection) Varying definitions, language, measurement instruments Form of accessible data Rapid change of technical data Fear (e.g., misuse, misunderstanding, adverse consequences)

10. C O B P 10 Why Do We Care About Data? Frequently drives solution strategy (e.g., study schedule, costs) Determines surrogate vice ideal MoMs Constrains viable scenarios Drives treatment of key factors; e.g., –Treatment of human factors –Addressing organizational issues Affects selection of tools

11. C O B P 11 What the Team Needs to Know About Data Data needs/data structure –Preferred –Necessary –Available Data accessibility –Ownership –Security issues –Costs (buy, collect, generate)

12. C O B P 12 Overall goals –Make data visible, accessible, and understandable across military organizations and beyond Obtaining data –Existing data: this entails finding, organizing, verifying, processing, and converting data –Non-available data (e.g., new concepts of operations) Tap SMEs, results of simulations,… Document via metadata Replace expert opinion with empirical data (e.g., experiments, operational data) ASAP Data conversion –Initial: vague, uncertain, incomplete, contradictory, soft –Desired: sharp, certain, complete, consistent, hard –Be explicit about how you converted the data ! Comments on Selected Data Challenges

13. C O B P 13 Selected Best Practices Near-term –Employ emerging “analytical baselines” (DODD 8260) (subsuming scenarios, concept of operations, integrated data) –Find and reuse data to the extent feasible –Use metadata to document key data actions (e.g., implement DoD Discovery Metadata Standard (DDMS)) -- and update appropriately! Longer-term –Establish a C2 Assessment Community of Interest –Align data processes and toolsets, as early as possible, with the C2 systems community (e.g., use same Information Resource Dictionary System) –Employ data engineering to gather, organize, convert, and VV&C available data

14. C O B P 14 Summary The data problem is complex and enormous … and increasing in both complexity and size! The community has taken significant initial steps to address the problem; e.g., –Issued new directives, instructions (e.g., DoDD 8260) –Created new organizations (e.g., Joint Analytic Data Management Steering Committee (JADMSC)) –Formulated a framework based on the concept of enterprise, community of interest, and private data –Promulgated new tools, standards (e.g., DDMS) However, in order to make further substantive improvements, we have to –Transform the culture (e.g., by implementing incentives, overcoming disincentives) –Educate & train the users, providers of data -- and the decisionmaker! –Implement new processes (e.g., work the metadata problem)