Statistics New Zealand’s Case Study ”Creating a New Business Model for a National Statistical Office if the 21 st Century” Craig Mitchell, Gary Dunnet, Matjaz Jug

Overview Introduction: organization, programme, strategy The Statistical Metadata Systems and the Statistical Cycle: description of the metainformation systems, overview of the process model, description of different metadata groups Statistical Metadata in each phase of the Statistical Cycle: metadata produced & used Systems and Design issues: IT architecture, tools, standards Organizational and cultural issues: user groups Lessons learned

Business model Transformation Strategy 1.A number of standard, generic end-to end processes for collection, analysis and dissemination of statistical data and information Includes statistical methods Covering business process life-cycle To enable statisticians to focus on data quality and implemented best practice methods, greater coordination and effective resource utilisation. 2.A disciplined approach to data and metadata management, using a standard information lifecycle 3.An agreed enterprise-wide technical architecture

BmTS & Metadata The Business Model Transformation Strategy (BmTS) is designing a metadata management strategy that ensures metadata: –fits into a metadata framework that can adequately describe all of Statistics New Zealand's data, and under the Official Statistics Strategy (OSS) the data of other agencies –documents all the stages of the statistical life cycle from conception to archiving and destruction –is centrally accessible –is automatically populated during the business process, where ever possible –is used to drive the business process –is easily accessible by all potential users –is populated and maintained by data creators –is managed centrally

A - Existing Metadata Issues metadata is not kept up to date metadata maintenance is considered a low priority metadata is not held in a consistent way relevant information is unavailable there is confusion about what metadata needs to be stored the existing metadata infrastructure is being under utilised there is a failure to meet the metadata needs of advanced data users it is difficult to find information unless you have some expertise or know it exists there is inconsistent use of classifications/terminology in some instances there is little information about data, where it came from, processes it has been under or even the question to which it relates

B - Target Metadata Principles metadata is centrally accessible metadata structure should be strongly linked to data metadata is shared between data sets content structure conforms to standards metadata is managed from end-to-end in the data life cycle. there is a registration process (workflow) associated with each metadata element capture metadata at source, automatically ensure the cost to producers is justified by the benefit to users metadata is considered active metadata is managed at as a high a level as is possible metadata is readily available and useable in the context of client's information needs (internal or external) track the use of some types of metadata (eg. classifications)

How to come from A to B? 1.Identified the key (10) components of our information model. 2.Service Oriented Architecture. 3.Developed Generic Business Process Model. 4.Development approach from ‘stove-pipes’ to ‘components’ and ‘core’ teams. 5.Governance – Architectural Reviews & Staged Funding Model. 6.Re-use of components.

10 Components within BmTS 2. Output Data Store Clean Data Aggregate Data 1. Input Data Store 3. Metadata Store Statistical Process Knowledge Base 9. Reference Data Stores 4. Analytical Environment 5. Information Portal 6. Transformations Raw Data 7. Respondent Management 8. Customer Management RADL Web Output Channels Multi-Modal Collection CURFS INFOS E-Form CAI Imaging Admin. Data Official Statistics System & Data Archive Summary Data ‘UR’ Data 10. Dashboard / Workflow

Time Series Store (& INFOS) Metadata Store (statistical, e.g. SIM) Reference Data Store (e.g. BF, CARS) Need Design/ Build CollectProcess Analyse Disseminate Software Register Document Register Management Information - HR & Finance Data Stores Statistics New Zealand Current Information Framework Generic Business Process ICS Store QMS, Ag HES etc. Web Store Range of information stores by subject area (silos)

Process Metadata Store (statistical/process/knowledge) Reference Data Store Need Design/ Build CollectAnalyse Disseminate Statistics New Zealand Future Information Framework Generic Business Process Raw Data TS ICS WEB Software Register Document Register Management Information - HR & Finance Data Stores Output Data Store (confidentialised copy of IDS - Physically separated) Clean Data Summary Data Input Data Store

CMF – gBPM Mapping CMF Lifecycle ModelStatistics NZ gBPM (sub-process level) 1 - survey planning and designNeed (sub-processes ) + Develop & Design (sub-processes ) 2 - survey preparationBuild (sub-processes ) + Collect (sub- process 4.1) 3 - Data collectionCollect (sub-processes ) 4 - Input processingCollect (sub-process 4.5) + Process (sub- processes ) 5 - Derivation, Estimation, Aggregation Process (sub-processes ) 6 - AnalysisAnalyse (sub-processes ) 7 - DisseminationDisseminate (sub-processes ) 8 - Post survey evaluationNot an explicit process, but seen as a vital feedback loop.

Metadata: End-to-End Need –capture requirements eg usage of data, quality requirements –access existing data element concept definitions to clarify requirements Design –capture constraints, basic dissemination plans eg products –capture design parameters that could be used to drive automated processes eg stratification –capture descriptive metadata about the collection - methodologies used –reuse or create required data definitions, questions, classifications Build –capture operational metadata about selection process eg number in each stratum –access design metadata to drive selection process Collect –capture metadata about the process –access procedural metadata about rules used to drive processes –capture metadata eg quality metrics

Metadata: End-to-End (2) Process –capture metadata about operation of processes –access procedural metadata, eg edit parameters –create and/or reuse derivation definitions and imputation parameters Analyse –capture metadata eg quality measures –access design parameters to drive estimation processes –capture information about quality assurance and sign-off of products –access definitional metadata to be used in creation of products Disseminate –capture operational metadata –access procedural metadata about customers –Needed to support Search, Acquire, Analyse (incl; integrate), Report –capture re-use requirements, including importance of data - fitness for purpose –Archive or Destruction - detail on length of data life cycle.

Metadata: End-to-End - Worked Example Question Text: “Are you employed?” Need –Concept discussed with users –Check International standards –Assess existing collections & questions Design –Design question text, answers & methodologies –Align with output variables (e.g. ILO classifications) –Data model, supported through meta-model –Develop Business Process Model – process & data / metadata flows Build –Concept Library – questions, answers & methods –‘Plug & Play’ methods, with parameters (metadata) the key –System of linkages (no hard-coding)

Metadata: End-to-End - Worked Example Question Text: “Do you live in Wellington?” Collect –Question, answers & methods rendered to questionnaire –Deliver respondents question –Confirm quality of concept Process –Draw questions, answers & methods from meta-store –Business logic drawn from ‘rules engine’ Analyse –Deliver question text, answers & methods to analyst –Search & Discover data, through metadata –Access knowledge-base (metadata) Disseminate –Deliver question text, answers & methods to user –Archive question text, answers & methods

Conceptual View of Metadata Anything related to data, but not dependent on data = metadata There are four types of metadata in the model: Conceptual (including contextual), Operational, Quality and Physical …defined by MetaNet

Metadata Implementation: Dimensional Model FACT Dimension Standard classifications Standard variables Survey Instruments Survey mode Standard data definition Standard questions

Metadata Dimensional Model FACT Standard classification s Survey Survey Instruments Survey mode Standard data definition Standard questions

Input Data Environment Metadata Architecture FACT Service layer Reference data Classifications INFORMATION PORTAL User access

Questions & Variables Fact definitions Collections & Instruments Respondents VersioningTime Dimensions Hiearchies Units of Interest

Goal: Overall Metadata Environment

Metadata: Recent Practical Experiences Generic data model – federated cluster design –Metadata the key –Corporately agreed dimensions –Data is integrateable, rather than integrated Blaise to Input Data Environment –Exporting Blaise metadata ‘Rules Engine’ –Based around s/sheet –Working with a workflow engine to improve (BPM based) IDE Metadata tool Currently s/sheet based Audience Model –Public, professional, technical – added system

SOA

Standards & Models - The MetaNet Reference Model TM Two Level Model based on: Concepts = basic ideas, core of model Characteristics = elements, attributes, make concepts unique Terms and descriptions can be adapted Concepts must stay the same Concepts should be distinct and consistent Concepts have hierarchy and relationships

Question 1 Question 2 Question 3 Question 2 Classifications Collection Questionaire A Questionaire B Collection Instance Fact definition 1 Fact definition 2 Fact definition 3 Fact definition 4 Do you live in Wellington? Person lives in Wellington Classification: CITY Category: WGTN Classification: NZ Island Category: NTH ISL Question 1 Fact definition 2 Classifications Question 3 Fact definition 4 Question 1 How old are you? What is your age? Age of person Eg. Census 2006 Eg. Census Frequency= 5 yearly

Defining Metadata Concepts: Example

How will we use MetaNet? 1.Use to guide the development of a Stats NZ model 2.Another model (SDMX) will be used for additional support in gaps 3.Provides the base for consistency across systems and frameworks 4.Will allow for better use and understanding of data 5.Will highlight duplications and gaps in current storage

Metainformation systems Concept Based Model SIM Other Metadata stored in: Business Frame Survey Systems BmTS components etc IDECARS Data Collections Variables Statistical Units Sample Design Classifications Categories Concordance Domain Value Collection Fact Classification Response

Metadata Users - External Government, Public, External Statisticans (incl. Intl Orgs)

Metadata Users - Internal –Statistical Analysts –IT Personnel (business analysts, IT designers & technical leads, developers, testers etc.) –Management –Data Managers / Custodians / Archivists –Statistical Methodologists –External Statisticians (researchers etc.) –Architects - data, process & application –Respondent Liaison –Survey Developers –Metadata and Interoperability Experts –Project Managers & Teams –IT Management –Product Development and Publishing –Information Customer Services

Lessons Learnt – Metadata Concepts Apart from 'basic' principles, metadata principles are quite difficult. To get a good understanding of and this makes communication of them even harder. Every-one has a view on what metadata they need - the list of metadata requirements / elements can be endless. Given the breadth of metadata - an incremental approach to the delivery of storage facilities is fundamental. Establish a metadata framework upon which discussions can be based that best fits your organisation - we have agreed on MetaNet, supplemented with SDMX.

Lessons Learnt – BPM To make data re-use a reality there is a need to go back to 1st principles, i.e. what is the concept behind the data item. Surprisingly it might be difficult for some subject matter areas to identify these 1st principles easily, particularly if the collection has been in existence for some time. Be prepared for survey-specific requirements: the BPM exercise is absolutely needed to define the common processes and identify potentially required survey-specific features.

Lessons Learnt – Implementation Without significant governance it is very easy to start with a generic service concept and yet still deliver a silo solution. The ongoing upgrade of all generic services is needed to avoid this. Expecting delivery of generic services from input / output specific projects leads to significant tensions, particularly in relation to added scope elements within fixed resource schedules. Delivery of business services at the same time as developing and delivering the underlying architecture services adds significant complexity to implementation.

Lessons Learnt – Implementation Well defined relationship between data and metadata is very important, the approach with direct connection between data element defined as statistical fact and metadata dimensions proved to be successful because we were able to test and utilize the concept before the (costly) development of metadata management systems.

Lessons Learnt – SOA The adoption and implementation of SOA as a Statistical Information Architecture requires a significant mind shift from data processing to enabling enterprise business processes through the delivery of enterprise services. Skilled resources, familiar with SOA concepts and application are very difficult to recruit, and equally difficult to grow.

Lessons Learnt – Governance The move from ‘silo systems’ to a BmTS type model is a major challenge that should not be under-estimated. Having an active Standards Governance Committee, made up of senior representatives from across the organisation (ours has the 3 DGSs on it), is a very useful thing to have in place. This forum provides an environment which standards can be discussed & agreed and the Committee can take on the role of the 'authority to answer to' if need be.

Lessons Learnt – Other There is a need to consider the audience of the metadata. Some metadata is better than no metadata - as long as it is of good quality. Do not expect to get it 100% right the very first time.

Questions?