InSPIRe Australian initiatives for standardising statistical processes and metadata Simon Wall Australian Bureau of Statistics December

2 Increasing cost & difficulty of acquiring data New competitors & changing expectations Rapid changes in the environment Competition for skilled resources Reducing budget Riding the big data wave Challenges facing NSIs

Reduce the cost and time of doing business Grow the business through new statistical products and services Deliver the first large scale digital Census (2016) on time, to budget and quality  while delivering Business As Usual. 3 Challenges facing ABS

Through large scale Innovation across the whole ABS We will: –radically transform the way we acquire, collate, use, reuse and disseminate statistical information By: –industrialising, modernising and reengineering our business processes –reengineering our statistical infrastructure and the way we manage information –develop capability needed to meet future needs –Collaborating with other international NSIs 4 How will we get there?

InSPIRe 5

What is InSPIRe? Infrastructure for Statistical Process and Information Management Re-engineering Building core infrastructure in the ABS for: –Business Process management –Information management SWM –Statistical Workflow Management System MRR –Metadata Registry and Repository 6

SWM The Statistical Workflow Management System An environment that will be used to develop and manage common business processes –These processes can drive and/or be used by tools and applications. 7

MRR The Metadata Registry and Repository consists of two parts: –Repository The centralised ‘bucket’ to store standards based metadata. –Registry The catalogue that lets you find out what is in the repository. 8

MRR Registers and stores a wide range of information, including: –Metadata (i.e. Classifications, Variables, etc) –Data – registered in MRR, stored in EDW –Process Definitions and Paradata – information to run a process –Process Metrics – information about a process that was run. 9

MRR The MRR –Enforces registration of the information, ensuring consistent documentation –Reads information in a range of standard supported metadata formats (initially DDI and SDMX) –Re-issues metadata in whatever standard format is required by a given process. Regardless of the format in which the metadata was originally created in. –This information is stored for use and later re-use by the metadata driven processes in SWM –Enables the searching and discovery of metadata for re-use. 10

Interfaces Processes and Systems will interact with the MRR through a web service interface that supports a controlled set of standards. –Currently limited to DDI 3.1 and SDMX

Benefits of InSPIRe Basis for greater automation and reuse of processes and metadata Faster to market –For individual collections –For building new collections Cheaper to market (long term) –For individual collections –For building new collections 13

Progress so far… Proof of Concept Project (2010/2011) –A group of simple use cases to prove the idea of the MRR and SWMs. Demonstrating: –Benefits –Potential functionality –What is technically achievable Building capability 14

Pathfinders Pathfinder Integration (July 2012) –Expanded on PoC –‘Pathfinder’ projects to produce further use cases for the MRR and SWMs. Not end-to-end processes, but isolated ‘snap shot’ projects to demonstrate the capabilities of InSPIRe. Four pathfinders chosen –Including web data capture (e-forms) and REEM (Remote Execution Environment for Microdata)

REEM and Web Data Capture Pathfinders Proved integration with real ABS systems Demonstrated the benefits of integration with InSPIRe, namely: –Re-use of processes (in SWM) –Standards based metadata (DDI in MRR) driving business tools –Automating business processes (in SWM)

Early Adopter Projects Projects which are positioning to integrate with InSPIRe within the next 12 months. –Includes projects focusing on administrative data, e- forms and the 2016 Census. Suitable candidates to prove the operation InSPIRe. –implementing processes in different phases of the GSBPM. 17

What we have achieved so far MRR –Design and review, mappers, shredders, database, automated generation of metadata types, basic search, registry model SWM –Environment setup, test cases with projects Capability Development –Best practice guidelines for implementing DDI, metadata content guidelines, InSPIRe integration information packs 18

What we still have to do MRR –Production version SWM –Further process definition, integration with other projects Metadata Authoring Environment –A mechanism for: Taking existing metadata content and registering it in the MRR Creating new metadata content via a content creation tool and registering it in the MRR 19

The future… An example 20

21 Survey Designer Retrieve Store Retrieve Store SWM