ABS Statistical Databases Session 6 Mark Viney Australian Bureau of Statistics 6 June 2007
INPUTTHRUPUTOUTPUT INPUTTHRUPUTOUTPUT INPUTTHRUPUTOUTPUT "Stove Pipe" approach
INPUT THRUPUT Standardised interface Standardised interface INPUT OUTPUT "Clearing-House" Approach Standardised interface OUTPUT IDW ABSIW
e-Census
e-Census 2006 Conducted 2006 Population Census with the option of electronic submission of responses ƒ drop-off/ pick up drop-off/mail back in 2011 10.2% of returns were electronic ƒ no edits incorporated into electronic form ƒ less visits to pick up paper forms ƒ less paper forms less scanning/repair
ABS Secure Deposit Box
Secure Deposit Box An externally facing database to allow respondents to lodge their raw data electronically ƒ Excel spreadsheet (essentailly replacing a paper form) ƒ Administrative datasets
ABS Statistical Databases ABS Input Data Warehouse (ABS IDW) ABS Information Warehouse (ABSIW)
ABS Input Data Warehouse (ABS IDW)
Input Data Warehouse Used as a repository for data as soon as it is entered into ABS computer systems ƒ Initially used for data received electronically ƒ Now used to load (and process) survey data
Input Data Warehouse Structure ƒ Star schema 1 fact table and several dimension tables each data cell is stored as 1 row in the fact table
Star Schema
ABS Input Data Warehouse - What it allows us to do Keep a historical record of what each cell was at every point in the processing ƒ Reason for the change ƒ when it changed ƒ who changed it ƒ change in value Ready access to both current and historical data
ABS Input Data Warehouse - What it allows us to do A data store for use with :- ƒ editing ƒ imputation ƒ winsorisation ƒ estimation Quick easy analysis and confrontation of data:- ƒ across time ƒ across dataitems ƒ across data sources
ABS Input Data Warehouse - Flow of Information
What we hope to achieve from IDW Reduced costs Improved data quality Tools to assist with management of data providers Better understanding of Editing processes ƒ Significance Editing One single source of microdata ƒ for all statistical collections Well managed and secure data storage
ABS Information Warehouse (ABSIW)
ABS Information Warehouse Need to make both data and metadata:- ƒ Visible ƒ Relatable ƒ Accessible ƒ Understandable ƒ Reliable ƒ Media Independent
ABS Information Warehouse Visible ƒ central known location Relatable ƒ across collections Accessible ƒ tools to allow extraction and manipulation
ABS Information Warehouse Understandable ƒ data fully described by metadata Reliable ƒ single source ƒ high availability Media Independent ƒ single source for outputs paper publications electronic releases ad - hoc requests
ABS Information Warehouse Define and manage metadata Load lightly aggregated data Validate data as compliant with metadata Manipulate data Produce statistical outputs Make data publicly available
ABS Information Warehouse - Flow of information Data from a collection Load info on how to categorize data Load info on what data items mean Load info about collection Load data to the ABSDB Closed DB Sign-off data to the ABSDB Open DB Disseminate output tables Derive ad- hoc client data requests Disseminate time series Processing System Information Warehouse PPW
ABS Information Warehouse - Define and Manage Metadata Interfaces to manage metadata ƒ load, amend, validate, extract ƒ dataitems,classifications, collections,datasets,publications Application Program Interfaces (API) to link with other systems/programs ƒ increasingly using XML
ABS Information Warehouse - Loading data Load data from major sources ƒ Input Data Warehouse ƒ SAS ƒ FAME ƒ SuperCROSS
ABS Information Warehouse - Generating New data Cubes Passing data through one or more steps to derive a new table ƒ aggregation ƒ drop dataitems ƒ calculate new items
ABS Information Warehouse - Other Manipulations Seasonal Adjustment ƒ SeasABS (X-11) Chain Volume Measures ƒ FAME (timeseries) Supertables Confidentialisation ƒ Disclosure Avoidance Analysis System
ABS Information Warehouse - Data Delivery Data combined with metadata Output formats created tailored to specific use ƒ spreadsheets ƒ timeseries ƒ supertables ƒ paper publications ƒ electronic release
ABS Information Warehouse - Public Release Make data available on an internally accessible database at a predetermined time (usually 11:30 am Canberra time) ƒ This data is then available to ABS Statistical Consultants to satisfy customer requests Feed data to website ƒ
ABS Website
National Data Network (NDN)
We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service Australian Bureau of Statistics
National Data Network Website that raises visibility of statistical data ƒ regardless of publishing agency A national platform for acquiring, sharing and integrating data relevant to policy and research in Australia
National Data Network One central website ƒ descriptions of data ƒ quality statement ƒ references to other data Several websites (Nodes) owned and maintained by other agencies
National Data Network
Current Focus ƒ Publish / Search / Acquire Future Focus ƒ Design / Capture / Process ƒ Analyse / Report
We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service Australian Bureau of Statistics
Questions?