1. Uncontrolled copy when printed I Burnell
22/09/2018
Brief to Quality of Service and Experience DWG – Development of a Data and Services Quality Model
Introduction
Approach
Quality Models
Requirements
Assessments
Conclusions
Questions
I Burnell
OFFICIAL
22 September 2018
© Crown copyright 2013 Dstl
© Crown copyright 2012 Dstl

2. So having provided some context, I would now like to look at the AGIS project.
The project was delivered by the GeoCore consortium which brought together leaders in GEOINT understanding, technology and research from both industry and academia.
It was led by Envitia (OGC Member) with Helyx (OGC Member) , QinetiQ and the University of Nottingham (OGC Member) all members of the core team. Other companies were brought in as required.
The AGIS project ran for 4 years in total and completed the 3rd phase of the research in March 2016.
OFFICIAL
22 September 2018
© Crown copyright 2013 Dstl
Copyright Envitia

3. Introduction
The quality of services (and intrinsically the data provided by those services) needs to be described and monitored in order for the Spatial Data Infrastructure (SDI) to be able to optimise its delivery of services.
‘Quality’ is an emerging subject for Defence with which it is relatively unfamiliar and there is a lack of consensus as to the contributing factors of spatial data and service quality and therefore no common definition:
Web service quality is inconsistently defined and implementation specific.
Data quality definitions focus on the production process and have limited value in the context of use.
There is no overarching policy specifically covering quality guidance, metrics and implementation in the context of SDI across the range of functions that impact on quality, i.e. from the supply of data at one end, through to search and discovery via web services at the other.
Aim of Research
Data and Service Quality and Optimisation built on previous AGIS research and focussed on:
Identifying quality use cases and requirements covering the range of activities that impact on ‘quality’
Defining quality factors and create a Data Quality Model (DQM) and Web Service Quality Model (WSQM) for use within the context of SDI
Investigating COTS tools and methodologies regarding management of data quality and web services and assess these against the requirements identified.
Copyright Envitia

4. Approach Define quality use cases and requirements:
Create use cases and requirements based on stakeholder input.
Investigate prioritised use cases using current examples of best practices and identifying suitable methodologies/approaches from stakeholders and academic input.
Create quality models:
Review current definitions of web service and data quality and create a draft data quality model (DQM) and web service quality model (WSQM).
Assess COTS tools and methodologies:
Assess planned tool implementation for service monitoring and optimisation under current work (i.e. LogRhythm).
Identify and assess relevant methodology for providing quality summaries to users (i.e. Content Maturity Model).
Identify and assess COTS data management tools against key features derived from DQM and requirements stages.
Define quality use cases and requirements:
Create use cases and requirements based on stakeholder input.
Investigate prioritised use cases using current examples of best practices and identifying suitable methodologies/approaches from stakeholders and academic input.
Create quality models:
Review current definitions of web service and data quality and create a draft data quality model (DQM) and web service quality model (WSQM).
Assess COTS tools and methodologies:
Assess planned tool implementation for service monitoring and optimisation under current work (i.e. LogRhythm).
Identify and assess relevant methodology for providing quality summaries to users (i.e. Content Maturity Model).
Identify and assess COTS data management tools against key features derived from DQM and requirements stages.
Copyright Envitia

5. Data Quality Model SDI DQM Data Quality Model (DQM)
Includes parameters to cover users’ perception of quality, based on GeoViQua.
SDI Data Quality Model (DQM) is standards based and includes three strands: contextual, quantitative and qualitative
SDI DQM
Quantitative
Contextual
Qualitative
Data Quality Model (DQM)
Includes parameters to cover users’ perception of quality, based on GeoViQua.
SDI Data Quality Model (DQM) is standards based and includes three strands: contextual, quantitative and qualitative
Copyright Envitia

6. Web Service Quality Model
Web Service Quality Model (WSQM)
A ‘core set’ of parameters was created by comparing W3C, INSPIRE, and MOD requirements
‘Core set’ was compared with OASIS WSQF and assessed for applicability to Defence
SDI WSQM is based on OASIS WFQF
Web Service Quality Model (WSQM)
A ‘core set’ of parameters was created by comparing W3C, INSPIRE, and other requirements
‘Core set’ was compared with OASIS WSQF and assessed for applicability to Defence
SDI WSQM is based on OASIS WFQF
Copyright Envitia

7. Requirements 1
Acquire or capture data
Data Manager is able to check for, and correct, data errors in a timely and consistent way (using automation whenever possible).
Data Manager should be able to populate metadata as efficiently as possible (using automation wherever possible) to ensure completeness and quality of metadata.
Data Manager should be able to measure, monitor and assess the quality of data and services in order to understand issues and trends.
Manage and maintain data quality
Manage data updates
Manage and optimise Web services
Consumer should be able to give qualitative feedback (e.g. ratings on dataset suitability) in order to augment subsequent user assessments of ‘quality’ during search and discovery.
Consumer should be able to view a summary of ‘quality’ in the most appropriate form for their purposes (e.g. on-the-fly messages, or a visual representation of quality), whilst ensuring that they do not suffer from ‘information overload’.
Web Manager should be able to measure, monitor and visualise the quality of data and services in order to understand issues and trends and optimise service
Publish or disseminate geospatial resources
Find Geospatial Resources
Data and service use cases and high level requirements were identified
Three use cases were prioritised for further investigation, as a result of input ‘Manage and maintain data quality’; ‘Manage and optimise web services’; and ‘Use geospatial resources’.
Use Geospatial Resources
Augment Business Intelligence
Copyright Envitia

8. Requirements 2
Case studies or approaches were identified for each prioritised use case to further refine requirements and identify gaps.
Data management case study identified the following shortfalls:
Limited re-use of quality measures
Limited use of corrective measures
Lack of quality monitoring
Review of Defence web service management found:
Lack of metrics gathered for monitoring and optimisation of service
Reliance on bespoke systems with no enterprise solution
Case studies or approaches were identified applicable for each:
Data management:
Shortfalls in current practice included: limited re-use of quality measures, limited corrective measures, lack of monitoring
Web service monitoring at Best:
Provided effective performance metrics and usage monitoring
The tools bespoke nature limits its re-use potential
Feedback mechanisms and quality summaries:
Aspiration is for user based comments and ratings
These could improve provision of more intelligent search and discovery
Review of feedback mechanisms and quality summaries found:
The aspiration is for user based comments and ratings
Copyright Envitia

9. Assessments
An RFI was sent to mainstream COTS data management tool vendors.
The tools were scored based on RFI responses, the use of software demos, background information and the results of industry assessment by Gartner Inc.
Key findings were:
The top three products emerged as Talend Data Integration and Data Quality, SAP Business Objects Business Intelligence and Business Objects Data Integrator, and Oracle Enterprise Data Quality.
PROS: quicker, more efficient QA process; proactive issue management by monitoring trends; consistent processes producing repeatable metrics; quality summaries that were easily disseminated and interpreted.
CONS: incompatibility with GIS formats; inability/limited ability to perform spatial analysis; out of the box functionality focussed on ‘customer’ type data.
The focus was on mainstream COTS Data Management tools. A Request for Information (RFI) was sent to twelve vendors, which incorporated user requirements and core functionality such as:
the ability to monitor quality trends, the ability to disseminate Enterprise wide quality information;
the ability to predefine and reuse quality rules;
the use of techniques such as standardisation and data linking for error detection.
Copyright Envitia

10. Conclusions User feedback:
Qualitative parameters, such as ‘user feedback’ are seen as high importance and are not included in quality specifications published by standards bodies.
The Content Maturity Model (CMM) describes an ideal methodology for communicating data quality to decision makers, as it uses a combination of producer and user centric input. However, it relies on user feedback mechanisms to be in place.
Data quality management:
ISO describes quantitative parameter metrics, but there is a requirement for a common or minimum set to be defined, which will impact on the tool required to capture them.
The COTS data quality management tools assessed all provide advanced techniques for quality assurance and can provide additional metrics to support the SDI DQM.
They all support quicker, more efficient workflows for quality management and the ability to communicate and monitor results effectively, however, ‘quick fixes’ could be provided using existing or free tools.
User feedback:
Qualitative parameters, such as ‘user feedback’ are seen as high importance and are not included in quality specifications published by standards bodies.
The Content Maturity Model (CMM) describes an ideal methodology for communicating data quality to decision makers, as it uses a combination of producer and user centric input. However, it relies on user feedback mechanisms to be in place.
Data quality management:
ISO describes quantitative parameter metrics, but there is a requirement for a common or minimum set to be defined, which will impact on the tool required to capture them.
The COTS data quality management tools assessed all provide advanced techniques for quality assurance and can provide additional metrics to support the SDI DQM.
They all support quicker, more efficient workflows for quality management and the ability to communicate and monitor results effectively, however, ‘quick fixes’ could be provided using existing or free tools.
OFFICIAL
22 September 2018
© Crown copyright 2013 Dstl
Copyright Envitia

11. Conclusions Web service optimisation:
Definitions of quality parameters for web services are highly variable, as is the method used to capture quantitative metrics (e.g. response time). These need to be defined in common implementation guidance.
The WSQM parameters cover functional and non-functional requirements of a service and their definitions and example metrics should form the basis for future web service requirement documents.
Implementation of LogRhythm software as part of SDI supports the provision of SDI WSQM metrics with the exception of the ‘business value quality’ parameter and configuration management within the ‘manageability quality’ parameter.
Web service optimisation:
Definitions of quality parameters for web services are highly variable, as is the method used to capture quantitative metrics (e.g. response time). These need to be defined in common implementation guidance.
The WSQM parameters cover functional and non-functional requirements of a service and their definitions and example metrics should form the basis for future web service requirement documents.
Implementation of LogRhythm software as part of SDI supports the provision of SDI WSQM metrics with the exception of the ‘business value quality’ parameter and configuration management within the ‘manageability quality’ parameter.
OFFICIAL
22 September 2018
© Crown copyright 2013 Dstl
Copyright Envitia

12. Questions
OFFICIAL
22 September 2018
© Crown copyright 2013 Dstl