Metadata Infrastructure and Standardisation in New Zealand Evelyn Wareham Manager, Information Management Statistics New Zealand METIS Working Session, May 2013

Overview Statistics New Zealand’s transformation Metadata Infrastructure programme What we’ve achieved What’s next Our approach Challenges and lessons learnt 7 May 2013 E Wareham METIS 2013

Statistics 2020 Statistics NZ is currently undertaking an organisation-wide programme of change, Statistics 2020 Te Kāpehu Whetū, to create the statistical system of the future. This programme has been developed to address challenges and opportunities in order to create a more dynamic, responsive, and sustainable organisation. The Strategic Plan 2010-2020 sets out what Statistics NZ is seeking to achieve over the next 10 years and how we plan to get there. The plan contains four key strategic priorities. Five key benefits will also be delivered, these are: 1. Continued supply of important and trusted statistics 2. An agile and responsive NSO able to respond to changing needs 3. Costs to government, businesses, and households minimised 4. Increased use of government data 5. Government has confidence that its investment in official statistics is value-for-money 7 May 2013 E Wareham METIS 2013

10 Year Transformation 160+ current projects Major infrastructure development Platforms replacing fragmented systems Standardisation Outwards orientation Culture change Traditionally business units across Statistics NZ stood alone and implemented systems for their needs. While the business units were operating well, they did not work together to create optimal efficiency for the organisation. As a result many of Statistics NZ's systems and software are now out-of-date, not supported by vendors, and only a few IT staff understand them. Resolving this system legacy problem is a major priority for the Board, as this issue could seriously affect Statistics NZ's reputation, plus its delivery and leadership roles in the official statistics system. The future plan is founded on effective and standardised technology, shared capability and common use platforms. With the legacy migration plan, the Platform roadmaps provide a staged transition from the old siloed architecture to the future state architecture allowing continued delivery of statistical outputs and expected time and cost savings. Standardisation is about developing statistical infrastructures and approaches to clusters of “like” statistics, and functions (e.g. collection and analysis) within the statistical production process… Standardisation is not a one size fits all approach. Rather it focuses on natural production clusters or integrated functions which have similar characteristics, thereby facilitating the introduction of a common, standardised approach to their management. 23/11/2018 E Wareham ALGIM 2012

Phases of Transformation 23/11/2018 E Wareham ALGIM 2012

Metadata Infrastructure Vision Information flows efficiently through the statistical business process, enabling more agile statistical production. Allowing us to be more customer-focused and respond more quickly to new information needs Making sure info resource required to support statistical production are available to be easily used and reused and where they’re not available, easily developed/created

We see an environment where … Less effort is needed to transport information across systems and adopt powerful new tools Less effort is needed to discover and understand existing data sources and information Our tools encourage standardisation and data (and metadata) re-use Good metadata management tools and systems Is a key foundational of this environment

Previous Situation Currently, information does not flow particularly efficiently between systems or across the statistical process We store key pieces of information that are by their nature structured and precisely defined – like questions, topic defintions, concepts, data item definitions- that are produced as part of the statistical process in Lotus Notes documents in DocONE libraries, in Excel spreadsheets, on our website on various pages and in various reports. And heaven help us if we want to re-use older data or use information from another part of the business - we often have to go on time-consuming treasure hunt searching through a variety of DocONE libraries and Notes databases or rely on the memory of key staff members. This means that when we want to use these pieces of information, for example, to load a processing system or a dissemination tool, etc. we have to manually massage that information into a new form (e.g. by cutting and pasting from Word or Lotus documents into an Excel spreadsheet) to meet the particular format requirements of a particular software packages or legacy database system or particular process.

Pain Points In terms of where this occurs, bottleneck before dissemination We often have to do this searching or manual re-shaping of information between phases of the statistical business process – particularly right before we disseminate data – and before we collect as well. So we merrily go along with production process and then we get to one of these points we have to do a bunch of work. effort is spent on these parts of the process

The Future

Look Reuse Create Information approach High – level approach to solving these problems - Our goal, then is to develop tools – software, standards, guidance, etc. – that allow us to better utilize our information to help us through the process, i.e. utilise key pieces of information that we naturally produce during the statistical process in an active way to help automate or drive some processes The organization should operate on the principle that– rather than making creation of new information resources in various shapes (documents, excel templates, load files for each system or process) the default, our systems should make re-using information the default So we want a model that make it easier to In the first place, LOOK for information or data RE-USE existing information or data And then only if something is not available, CREATE new a resources This will both save us money and support standardisation

Systems And looking in a bit more depth at the individual systems or tools that may be part of that metadata box or layer, these are the systems that would comprise such a layer. We’ve already built 1 & 2

Progress to date 2011-2012 April 2012: Colectica deployed, training starts June 2012: CARS, Micro-economic & Household platform metadata surfaced in Colectica June 2012: Standards and guidelines developed 2012-2013 Oct 2012: Classification System development begins Oct 2012-2013: Data custodians update information in Colectica October 2012: RPD process happens via Colectica May 2013: OSS metadata improvement pilot conducted May 2013: Colectica Portal available on external website

Improvements to come 2013-2014 Apr 2014: Classification Management system developed Dec2013: Services to allow other systems to access metadata developed 2014-2020 Question library? Concept library? Metadata standardisation

Coming Soon! 23/11/2018 E Wareham ALGIM 2012

Our Approach Series of stages: No big bang Partnerships with innovative vendors International collaboration Standardisation 80:20, not perfection Culture change, not just systems

Challenges and Lessons Learnt Selling the benefits – these may be to others, not to those doing the work Tailoring the message – do they need to understand DDI, SDMX and GSIM? Continuity and getting the timing right – must connect to legacy systems, but look to the future Can’t do it alone – partnerships are critical to success 7 May 2013 E Wareham METIS 2013

evelyn.wareham@stats.govt.nz www.stats.govt.nz Questions? evelyn.wareham@stats.govt.nz www.stats.govt.nz