Elaborating on the Business Architecture of SN Robbert Renssen Statistics Netherlands Standard Process Steps

Outline of the Presentation 1.Present situation i.IAF ii.The Business Architecture of SN iii.Relation with GSBPM 2.New Ideas i.Why ii.Outline

Standard Process Steps 2 BusinessInformation Systems Technical Infrastructure Contextual Conceptual Logical Physical Technical facilities Functionality & Usability Information needsBusiness concepts Hardware & System Software Components Application Components Communication patterns Organisational Structure & Procedures Products & Tools Bespoke Software & Settings Information Storage and Exchange Operations IAF Vision & Strategy Principles Standards Trends & developments Governance Security

Business Architecture SN 3 Business Architecture of SN Statistics Production Chain Management Design Central Storage of Steady States

Business Architecture SN 4 Relation GSBPM GSBPM 4 till 7 Not covered GSBPM 1 till 3 Central Storage of Steady States Not covered

Standard Process Steps 5 External supplierExternal customer Input base Micro base Output base Stat base Data collection Data processing Data dissemination Pre- input base Post- output base Internal users of selections of steady states Internal suppliers of steady states New Ideas

Standard Process Steps 6 Why new ideas about data processing Top-down (general) Courses on Design and Implementation of Surveys, see e.g. Willeboordse Generic Statistical Business Process Model (GSBPM) Bottom-up (specific) Process descriptions in view Business analysis, Business architecuture, VIR Gap between top-down and botton-up Top-down  driven by statistical methodology that underlies a process Bottom-up  driven by (physical) activities that are needed to carry out a process Something is missing: Ideas about standard process steps try to complete the picture

Standard Process Steps 7 Steady state D D Outline of the ideas Real World Property: Conceptual Variables Measured Property: Operational Variables Making an operational set

Standard Process Steps 8 Making an Operational Set under Closer Scrutiny (Example) Primary data collection Secundary data collection Matching Estimating Steady state B Steady state C Steady state D A B Steady state A Real World PropertyMeasured Property

Standard Process Steps 9 Closer Scrutiny; Matching, what is going on? Two datasets concerning the same population Having common unique identifyable variables Are put together (matched) on the criterium of identical ‘keys’. In reality, two datasets that are to be matched do not concern (exactly) the same population do not have (exactly) common unique identifyable variables have missing data and measurement errors in these (vitually) common unique identifyable variables

Standard Process Steps 10 We Need Some Preparation To obtain the same population, we need to Derive units, Delineate populations, … To obtain common unique identifyable variables, we need to Derive variables, Code variables, … To handle missing data or errors in the identifyable variables, we need value checks, … Like Matching, the preparing steps Derive units, Derive Variables, Code Variables, are elementary (statistical) functions As a combination, they work together for matching purposes

Standard Process Steps 11 Matching Reconsidered Our idea distinguishes between An (elementary) statistical function such as matching that can only be applied when the pre-conditions with respect to design of the inputs are fulfilled A statistical function flow, e.g. on behalf of matching, that consists of a flow of statistical functions and decision rules Characteristics of a statistical function: Based on a (statistical) method or subject matter knowledge Managed by so-called method rules, i.e. specifications of statistical methods or subject matter knowledge Characteristics of a statistical function flow: Consists of one or more statistical functions and decision rules On behalf of a statistical target (reason of designing this flow)

Standard Process Steps 12 Statistical Targets: Why Matching Data Sets To enrich a dataset with additional variables To observe and estimate relations (e.g. cross tabulations) As a preperation step to delineate populations by means of the additional variables As a preperation step to improve data (editing) with additional auxiliary information As a preperation step to estimate population totals with additional auxiliary information … To measure under- or overcoverage wrt the instance of a population To correct for under- or overcovarage wrt the instance of a population

Standard Process Steps 13 Design of Statistical Process Design of Process Flow Design of Statistical Function Flow Statistical Function data quality (statistical method) Decision Function Planning Statistical Function real-world property (statistical methode) Set of Conceptual Attribute Variables Statistical Target Set of Physical Attribute Variables Standard Non-Statistical Function Statistical Strategy Statistical Methods Completing the picture: an impression

Standard Process Steps 14 Thank You

Standard Process Steps 15 Design of Statistical Process Design of Process Flow (Activities) Design of Statistical Function Flow Statistical Function data quality (statistical method) Decision Function Planning Statistical Function real-world property (statistical methode) Set of Conceptual Attribute Variables Statistical Target Set of Physical Attribute Variables Set of Conceptual Quality Indicators Set of Logical Quality Indicators Set of Physical Quality Indicators Standard Non-Statistical Function Statistical Strategy Set of Logical Attribute Variables Statistical Function

Standard Process Steps 16 Hierarchy of Statistical Targets Relevant and Published Statistical Information (Mission) Effective and Efficient (Operational) Management Continuity of the Organisation Collecting statistical data Processing statistical data Analysing statistical data Disseminating statistical information wrt Collecting statistical data wrt Processing statistical data wrt Analysing statistical data wrt Disseminating statistical information Steady State: Micro data and quality meta data Steady State Macro data and quality meta data Derived Units Delineated Populations Inspected Coverage Corrected Coverage Coded Units Nested Value Domains Derived Variables Matched Variables (exact) Inspected Measurement Errors Corrected Measurement Errors Estimated Population Parameters Reconciled Estimates Estimated margins of Error Improved Estimates Auxiliary Micro data Auxiliary Macro data Derived Units Delineated Populations Sampled Populations Coded Units Nestes Value Domains Derived Variables Matched Variables Estimated Population Parameters Reconciled Estimates Quality Control for (operational) management Inspected Coverage Inspected Measurement Errors Estimated Margins of Error

Standard Process Steps 17 Examples of Functions (statistical / non-statistical) Function to Match variables Automatic based on a matching criterion Automatic based on a ‘matching table’ Function to Code Variables Visual (by hand), Automatic based on a ‘knowledge table’ Function to impute data By hand / Visual, Automatic based on hot deck models, Automatic based on regression models, Automatic based on ‘systematic’ edit rules Automatic based on algorithms Function to estimate population totals Weighting based regression models, Small Domein Estimation based on mixed models, Function to Check Errors By hand / visual using (nested) cell estimates Automatic based on conceptual domains Automatic bases on conceptuel restrictions (among others: sum restrictions), Automatic based on typical systematic errors Automatic based on robust estimates Function to Reconcile Estimates By hand / Visual, Lagrange restrictions, Bayesian model General Statistical Functions (statistical analyses, variances, means, medians, …) … Function to Recode Variables Automatic based on a recoding table Function to Transform Formats General Functions to manipulate data (aggregate, select, sum, multiply, substract, divide, sort, merge …) General functions to manipulate matrices (aggregate, multiply, invert, …) …

Standard Process Steps 18 Example of a Statistical Function Flow Function for Delineating populations Function for Checking and Correction (systematic) errors Function for detecting influential errors Function for Checking and Correcting errors (automatic) Function for Checking and Correcting errors (visual, by hand) Function to Estimate population parameters (weighting and aggregating) Function for Checking errors (macro level; visual, by hand) Function for Checking and correcting errors (micro level, by hand) Function for Deliniating Populations Function for Checking and Correcting (systematic) errors Function for checking and correcting errors (automatic) Function to Estimate population parameters (weighting and aggregating) Function for matching variables (t-1) Function for matching variables (population frame) For delineating populations For checking and correcting errors For checking errors (macro level)

Standard Process Steps 19 Terminology: Mapping on IAF Business Function Business Service Logical Conceptual B(usiness) Business Process Model Business Activity Business Object Business Inter- action Model Statistical Target Flow of Functions B(usiness) Statistical Function Variables, Rule Sets Input-Output Matrix? I(nformation) Information Object Information Interaction Model Information Service Physical Operational Process Model Flow of Activities