VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote

VARTAN – Project Objectives VARTAN is concerned with... Definition of Validation Reporting Templates in accordance with European Operational Concept Validation Methodology (E-OCVM) Analysis of E-OCVM Refinement of Life Cycle Model and Case-based View Context of templates in Life Cycle Model Study for automated transfer of template contents into the Validation Data Repository (VDR)

VARTAN – Analysis Issues Performance expressed in structured way? –Key Performance Areas (PRC/ICAO) –Key Performance Indicators Evidence with right level of quality? –Validity and accuracy of results (and objectives) Right levels of detail at different decision-making levels? –Strategy, programme, project, and experiment level Information needs of all decision makers addressed? –Airlines, service providers etc. E-OCVM comprehensible for less proficient user? –Simplify methodology for different E-OCVM views

VARTAN – Results Overview Suggested (or similar) refinements are envisaged for implementation in the updated methodology: Improve comprehensibility of Life Cycle Phase Model Refine case-building process description Describe and give examples of possible stakeholder needs and associated stakeholder analysis issues Consider confidence in results Activities for preparation of document standards with automation: Build templates at different planning levels and determine triggers for reporting Provide automation example for experiment-level validation reporting template Describe process for working with result templates

VARTAN – Suggested Life Cycle Refinements

VARTAN – Suggested Case-building Refinements From KPA and KPI to case definition: which cases are necessary for decision making? Interrelationships between defined areas (cases) Differences in focus from V1 to V3; from identification of showstoppers (V1) to implementation decision (V3) Discern Verification and Validation (especially in V2) Different levels of detail in cases: what level of detail is necessary for decision making per lifecycle phase?

VARTAN – Proposals for Stakeholder Needs Conflicting stakeholder objectives/goals or interests –Goal balancing –Harmonisation of concepts and ideas Different stakeholder information needs –Different levels of reporting Different levels of quality (validity/accuracy) of results –Use of standard methods to increase (or find a common level of) validity

VARTAN – General Template Considerations Pre-defined introductory part (background, context, scope) Summary of planning document Differences between planning and actual achievements Report confidence in results Objective should be related to a certain KPA Correlation between results / conclusions / recommendations and objectives (from high-level to low-level to hypotheses)

VARTAN – Planning and Reporting Levels Experiment Level Strategy Level Programme Level Project Level

VARTAN – Planning and Reporting Levels

relevant VDR information

VARTAN – Unexpected Results Validation Reporting Triggers (Exceptional triggers) Exceptional situations with unexpected results or events might lead to a re-evaluation of the current activity Need to report to higher levels immediately and therefore trigger a special report Important: there must be a way to stop the validation cycle and reconsider the complete approach Generally, such reporting is part of a good management approach (not part of E-OCVM guideline structure)

VARTAN – Validation Reporting Process (1/3)

VARTAN – Template Life Cycle Empty Template Accepted Template Prepared Template Empty Template Accepted Template Prepared Template Filled-in Template Received Template Approved Template Filled-in Template Received Template Approved Template Author accepts data VDR Data Collector adds data Author doesnot accept data VDR Administrator approves template VDR Administrator rejects template Author transmits template Author fills in template Author updates template

VARTAN – Validation Reporting Process (2/3)

VARTAN – Template Life Cycle Empty Template Accepted Template Prepared Template Empty Template Accepted Template Prepared Template Filled-in Template Received Template Approved Template Filled-in Template Received Template Approved Template Author accepts data VDR Data Collector adds data Author doesnot accept data VDR Administrator approves template VDR Administrator rejects template Author transmits template Author fills in template Author updates template

VARTAN – Validation Reporting Process (3/3) Validation Expert VDR Administrator TCP/IP HTTP Web Services – SOAP (XML) Apache Axis Java Servlet VDR (Oracle) VDR Interface SQL Statements

VARTAN – Template Life Cycle Empty Template Accepted Template Prepared Template Empty Template Accepted Template Prepared Template Filled-in Template Received Template Approved Template Filled-in Template Received Template Approved Template Author accepts data VDR Data Collector adds data Author doesnot accept data VDR Administrator approves template VDR Administrator rejects template Author transmits template Author fills in template Author updates template

VARTAN – Template Contents: Help Text

VARTAN – Template Contents: Introduction 1. Introduction 1.1Project/Experiment Background and Context 1.2Concept Overview 1.3Purpose of Document 1.4Scope and Structure of Document 1.5References 1.6Acronyms

VARTAN – Template Contents: Project Level Strategy and Planning 2. Summary of Project Strategy and Planning 2.1Stakeholder Requirements 2.2Supporting Cases 2.3Expected Project Outcomes and Objectives 2.3.1Description of Expected Project Outcomes 2.3.2Summary of Expected Project Outcomes 2.3.3Description of Project Objectives 2.3.4Summary of Project Objectives 2.4Decision Processes in Experiment Definition 2.5Experiment Context and Planning 2.6Analysis Strategy

VARTAN – Template Contents: Project Level Conduct and Results 3. Conduct of Validation Project 3.1Executed Experiments 3.2Deviations from the Planning 4. Results of Experiments 4.1Results of Different Experiments per Key Performance Area 4.2Confidence in Results of Different Experiments Quality of Results of Different Experiments Significance of Results of Different Experiments 4.3Unexpected Project Results 5. Consolidation of Project Results 5.1Consequences of Confidence in Experiment Results 5.2Consequences of Experiment Results for Project Objectives Interpretation of Results with respect to High-level Objectives Interpretation of Results with respect to Key Performance Areas 6. Conclusions and Recommendations for Programme Level 6.1Conclusions 6.2 Recommendations

VARTAN – Template Contents: Experiment Level Strategy and Planning 2. Summary of Experiment and Strategy Planning 2.1Expected Experiment Outcomes, Objectives and Hypotheses 2.1.1Description of Expected Experiment Outcomes 2.1.2Summary of Expected Experiment Outcomes 2.1.3Description of Experiment Objectives 2.1.4Summary of Experiment Objectives 2.1.5Description of Experiment Hypotheses 2.1.6Summary of Experiment Hypotheses 2.2Choice of Metrics and Measurements 2.3Choice of Methods and Techniques 2.4Scenario Specifications 2.5Experimental Variables and Design

VARTAN – Template Contents: Experiment Level Conduct and Results 3. Conduct of Validation Exercise Runs 3.1Experiment Preparation 3.2Executed Experiment Schedule 3.3Deviations from the Planning 4. Experiment Results 4.1Measured Experiment Results 4.2Confidence in Experiment Results Quality of Results of Experiment Significance of Results of Experiment 4.3Unexpected Experiment Results 5. Analysis of Experiment Outcomes 5.1Analysis of Outcomes on the Basis of Determined Hypotheses 5.2Analysis of Consequences of Outcomes for Experiment Objectives 6. Conclusions and Recommendations for Project Level 6.1Conclusions 6.2 Recommendations

VARTAN - Template Automation: Basic Information

VARTAN - Template Automation: Standard Structure

VARTAN - Template Automation: Entry Fields

VARTAN – Outlook into the Future Data definition and requirements for (static) document data and pre-formatted data Data structures for VDR entry including mutual relations Encryption standards for usernames and passwords Data protection modes and protected sections List of all property fields, bookmarks and macros in use by the automated solution to avoid conflicts Define rollback mechanisms for all automated changes Templates available at VALUG website: Website offers contact information and is available for feedback on methodology and standard methods

Questions