1. ABS Statistical Databases Session 6 Mark Viney Australian Bureau of Statistics 6 June 2007

2. INPUTTHRUPUTOUTPUT INPUTTHRUPUTOUTPUT INPUTTHRUPUTOUTPUT "Stove Pipe" approach

3. INPUT THRUPUT Standardised interface Standardised interface INPUT OUTPUT "Clearing-House" Approach Standardised interface OUTPUT IDW ABSIW

4. e-Census

5. e-Census 2006  Conducted 2006 Population Census with the option of electronic submission of responses ƒ drop-off/ pick up  drop-off/mail back in 2011  10.2% of returns were electronic ƒ no edits incorporated into electronic form ƒ less visits to pick up paper forms ƒ less paper forms  less scanning/repair

6. ABS Secure Deposit Box

7. Secure Deposit Box  An externally facing database to allow respondents to lodge their raw data electronically ƒ Excel spreadsheet (essentailly replacing a paper form) ƒ Administrative datasets

8. ABS Statistical Databases  ABS Input Data Warehouse (ABS IDW)  ABS Information Warehouse (ABSIW)

9. ABS Input Data Warehouse (ABS IDW)

10. Input Data Warehouse  Used as a repository for data as soon as it is entered into ABS computer systems ƒ Initially used for data received electronically ƒ Now used to load (and process) survey data

11. Input Data Warehouse  Structure ƒ Star schema  1 fact table and several dimension tables  each data cell is stored as 1 row in the fact table

12. Star Schema

13. ABS Input Data Warehouse - What it allows us to do  Keep a historical record of what each cell was at every point in the processing ƒ Reason for the change ƒ when it changed ƒ who changed it ƒ change in value  Ready access to both current and historical data

14. ABS Input Data Warehouse - What it allows us to do  A data store for use with :- ƒ editing ƒ imputation ƒ winsorisation ƒ estimation  Quick easy analysis and confrontation of data:- ƒ across time ƒ across dataitems ƒ across data sources

15. ABS Input Data Warehouse - Flow of Information

16. What we hope to achieve from IDW  Reduced costs  Improved data quality  Tools to assist with management of data providers  Better understanding of Editing processes ƒ Significance Editing  One single source of microdata ƒ for all statistical collections  Well managed and secure data storage

17. ABS Information Warehouse (ABSIW)

18. ABS Information Warehouse  Need to make both data and metadata:- ƒ Visible ƒ Relatable ƒ Accessible ƒ Understandable ƒ Reliable ƒ Media Independent

19. ABS Information Warehouse  Visible ƒ central known location  Relatable ƒ across collections  Accessible ƒ tools to allow extraction and manipulation

20. ABS Information Warehouse  Understandable ƒ data fully described by metadata  Reliable ƒ single source ƒ high availability  Media Independent ƒ single source for outputs  paper publications  electronic releases  ad - hoc requests

21. ABS Information Warehouse  Define and manage metadata  Load lightly aggregated data  Validate data as compliant with metadata  Manipulate data  Produce statistical outputs  Make data publicly available

22. ABS Information Warehouse - Flow of information Data from a collection Load info on how to categorize data Load info on what data items mean Load info about collection Load data to the ABSDB Closed DB Sign-off data to the ABSDB Open DB Disseminate output tables Derive ad- hoc client data requests Disseminate time series Processing System Information Warehouse PPW

23. ABS Information Warehouse - Define and Manage Metadata  Interfaces to manage metadata ƒ load, amend, validate, extract ƒ dataitems,classifications, collections,datasets,publications  Application Program Interfaces (API) to link with other systems/programs ƒ increasingly using XML

24. ABS Information Warehouse - Loading data  Load data from major sources ƒ Input Data Warehouse ƒ SAS ƒ FAME ƒ SuperCROSS

25. ABS Information Warehouse - Generating New data Cubes  Passing data through one or more steps to derive a new table ƒ aggregation ƒ drop dataitems ƒ calculate new items

26. ABS Information Warehouse - Other Manipulations  Seasonal Adjustment ƒ SeasABS (X-11)  Chain Volume Measures ƒ FAME (timeseries)  Supertables  Confidentialisation ƒ Disclosure Avoidance Analysis System

27. ABS Information Warehouse - Data Delivery  Data combined with metadata  Output formats created tailored to specific use ƒ spreadsheets ƒ timeseries ƒ supertables ƒ paper publications ƒ electronic release

28. ABS Information Warehouse - Public Release  Make data available on an internally accessible database at a predetermined time (usually 11:30 am Canberra time) ƒ This data is then available to ABS Statistical Consultants to satisfy customer requests  Feed data to website ƒ www.abs.gov.au

29. ABS Website www.abs.gov.au

31. National Data Network (NDN) www.nationaldatanetwork.org

32. We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service Australian Bureau of Statistics

33. National Data Network  Website that raises visibility of statistical data ƒ regardless of publishing agency A national platform for acquiring, sharing and integrating data relevant to policy and research in Australia

34. National Data Network  One central website ƒ descriptions of data ƒ quality statement ƒ references to other data  Several websites (Nodes) owned and maintained by other agencies www.nationaldatanetwork.org

36. National Data Network

37.  Current Focus ƒ Publish / Search / Acquire  Future Focus ƒ Design / Capture / Process ƒ Analyse / Report

38. We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service Australian Bureau of Statistics

39. Questions?