Alan Robinson / Mark Taylor / Terry Hooper Dstl OA Data Management An Overview of the Current Status of Data Management in Dstl Alan Robinson / Mark Taylor / Terry Hooper Dstl Explain who I am what I do MOD makes decisions CDA does analysis -- we don’t do modelling as an end in itself I am responsible for development of tools that CDA uses to do analysis Have management and processes in place to achieve this.

Dstl Data Management - ISMOR 20 Current Status - General Alan Robinson/Terry Hooper - Land OA Toolset Mark Taylor Data Quality Score Pete Bailey Collection & Application of Exercise C2 Data Mark Ashforth Data Management Principles: UK Perception of MORS W/Shop Mark Taylor/Terry Hooper Geoff Sherwood(Adaptive Ltd) 06 December 201806 December 2018

What do we mean by data management? Maintaining a body of required data for use by analysts Facilitating the re-use of data where appropriate Providing an audit trail - source, context and validation Ensuring consistency and coherence across applications Identifying need for data-collection or -generation Establishing common standards for provision/use -where appropriate, encourage common tools & methods 06 December 201806 December 2018

Why do we need effective data management ? Effective data management will: Improve the efficiency and validity of study work inter alia leading to reduced study costs. Maintain consistency of data across studies and thus enhance coherence between them. Through improved credibility, increase customer confidence in the data used and in the analysis that uses them. 06 December 201806 December 2018

Why is it difficult? - (1) Value of data management not well appreciated Tendency to “muddle through” data difficulties Failure to recognise potential spend-to-save No sound assessment of the status quo against which to assess improvements and likely value for money Co-ordination across data management initiatives not always achieved Partly due to lack of customer co-ordination Partly due to labelling - not all relevant work is labelled as data management; data related activity often occurs as part of other projects 06 December 201806 December 2018

Why is it difficult? - (2) Managing data effectively requires an understanding of context OA pulls together military, technical and other inputs often without the benefit of common terminology and data standards There are usually interdependencies between data items OA models and ‘softer’ methods have grown in complexity and been brought to bear on (even) more challenging problems e.g. C2, OOTW and Strategic Analysis. 06 December 201806 December 2018

Why is it difficult? - (3) Toolset support has often not been as effective as desired: Often tightly coupled to a target model Some false starts Advances that should enable more generic toolset support only recently becoming available and mature e.g. XML for data transformation 06 December 201806 December 2018

Progress to date Progress has been made with necessary process, data sources and toolset support, for example: Dstl data management strategy and associated handbook exists GENeric Information Environment (GENIE) toolset under ARP15 Network of data custodians captured via an OA Data Directory …however, improvements in data management have not been as significant as anticipated 06 December 201806 December 2018

Where next?(1) Managing “data with context” (information ?) has proven harder than originally expected Progress requires both: “Research” activity - to address research, development and transition-to-study-use issues “Management” activity - to ensure appropriate processes and culture are in place 06 December 201806 December 2018

Where next? (2) “Research” activity - “Management” activity - Needs customer support and funding to address research, development and transition-to-study-use issues “Management” activity - Needs Dstl action with appropriate customer support to ensure appropriate processes and culture are in place Neither strand alone will suffice 06 December 201806 December 2018

Recommended areas to be tackled M - Develop a network of responsible data co-ordinators M - Overhaul and re-issue Data Management Handbook R - Develop appropriate toolset support, using generic methods wherever practicable R - Improve methods for setting data in context R - Investigate and, if appropriate, adopt data standards M/R - Establish method of assessing benefit of data management M - Improve co-ordination in both customer and Dstl 06 December 201806 December 2018

ARP15 Data Management Funded by ARP15 over the past 3-4 years Previous work has focused on the development of tools FY03/04 programme on focused on: Management of scenario data Continued rollout of existing tools Implementation of a data management team within LSD Continuation of international links Support to wider Dstl and MoD data management activities 06 December 201806 December 2018

ARP15 Data Management Framework System Data (GDMS) Terrain (DGIA) Manoeuvre Compendium of Scenarios Capability Database (GDMS ) Input Tools XML Static File Exchange Formats ORBAT & (SID) Chunk SID Current GENIE Tools Model Files Model Run Management System Checking Runs & Output Post Run Analysis Model Boundary Deployment JCD SAG Study data Models Data Transforms Input Data Sub (e.g. ARP 07 Database) Key Manoeuvre Threat Manual SAG & JCD Process 06 December 201806 December 2018

Scenario Data ARP15 Manoeuvre Compendium of Scenarios: Contains scenario information at Formation level and below Has links to the electronic SAG book and existing JCD scenario descriptions Existing lower level scenario documentation grouped by SAG setting and study Data accessed via a web page on the Dstl secret network Scenario documents can be download from the compendium in electronic format 06 December 201806 December 2018

Toolset Development Work has been undertaken within ARP15 over the past 3-4 years to provide a set of generic tools to support data management for individual models and across capability areas Known as the GENeric Information Environment (GENIE) Initially focused on the management of model input data Can be extended to cover other areas, e.g. model run management and post run analysis GENIE is transitioning from development into routine use 06 December 201806 December 2018

GENIE The current GENIE toolset consists of three applications: Generic Database Management System (GDMS) Stores static data in a modified Oracle database COTS data transformation tools Mercator and XMLSpy Scenario Interactive Display (SID) ORBAT definition and deployment on to the terrain Model input data entry Generation of model terrain databases Replay of model runs 06 December 201806 December 2018

GDMS Overview Built on Oracle COTS database software Multi-user Designed to store large volumes of data Data defined objects and templates Forms based front end Primary role is to support capability databases rather than model specific data 06 December 201806 December 2018

Data Transformation Data transformation is needed to: Change the format of data Aggregate or disaggregate data Data is transformed using: Mercator COTS product Bespoke tools based on Excel, VBA, and high level programming languages e.g. C++ Experiments with XML have shown its potential, despite limited mathematical capabilities 06 December 201806 December 2018

SID Functionality Provides the following functionality: Model data dictionary specification ORBAT definition and deployment of units onto terrain Cut and paste functionality to assist with ORBAT set-up Propagation of changes to data dictionary to extant ORBAT files Generation of terrain data for models Generation of Line-of-sight statistics from digital terrain databases Replay of model runs 06 December 201806 December 2018

Data dictionary / ORBAT generation 06 December 201806 December 2018

Terrain Generation Can be generated at user defined resolution DTED, DFAD, DCW and VMAP level 0 data sources can be used Culture / features can be painted on to terrain and saved 06 December 201806 December 2018

Unit deployment 06 December 201806 December 2018

Replay Requires unit ORBAT to be set-up in SID Potentially any model output can be replayed User can define colours and details viewed 06 December 201806 December 2018

SID current status Functionality available in version 2 has been improved to cover core areas noted Tool has been thoroughly beta tested User guide has been completed Tool is ready for release Users: Tool is being used actively by WISE, FCM and CLASS teams CLASS will soon start using SID for static data entry 06 December 201806 December 2018

Conclusions Data management within ARP15 has proved a far harder problem to solve than originally expected The real need is to manage information, i.e. data and its associated context Results in increased technical and cultural complexity Data management tools only provide part of the solution Requirement is to define a workable data management process that operates in conjunction with the tools Will require support from MoD and Dstl stakeholders, ARP15 and Dstl LSD 06 December 201806 December 2018