Standard data entry & validation system in HCSO (ADEL) Erzsébet Kómár 3. May IT Department

Dissemination Database Data Warehouse Production Database Data for Survey Control Registers METADATAMETADATA Metadata- maintenance (META) Register- maintenance ( business reg. address reg, etc.) Survey Control (GESA, LAKOS) Data Collection (KSHXML) Data Processing (EAR) Dissemination (STATINFO) The main subsystems and concerning database in HCSO Data entry and validation (ADEL,BLUMEN)

Data entry & validation – past, present In regional offices, in-house activity 1982 – pdp/1140 computes, different validation programs 1982 – pdp/1140 computes, different validation programs 1993 – PC – nothing changed 1993 – PC – nothing changed 1999 – reorganized the process – Oracle DB, tools 1999 – reorganized the process – Oracle DB, tools unified user interface, same functions-same solutions, survey&term independent system, meta driven modules, central DB -> in-house developed application framework (ADEL) 2000 – introduction 1 survey 2000 – introduction 1 survey 2001 – 28 surveys 2001 – 28 surveys … contiguous development, new functions 2010 – 100 surveys 2010 – 100 surveys

Users and main functions Statisticians in the centre (subject matter department) Statisticians in the centre (subject matter department) design the questionnaire, select enterprises, define validation rules (in predefined format) design the questionnaire, select enterprises, define validation rules (in predefined format) IT experts in the centre IT experts in the centre provide IT background, define META information, design DB structure, grant/revoke roles, develop validation rules and data entry application provide IT background, define META information, design DB structure, grant/revoke roles, develop validation rules and data entry application Statisticians in the regional offices Statisticians in the regional offices keep contact with enterprises, collect questionnaires, update response information in survey control system, data entry, validate (paper and e-questionnaire) and correct data with ADEL framework keep contact with enterprises, collect questionnaires, update response information in survey control system, data entry, validate (paper and e-questionnaire) and correct data with ADEL framework Statisticians in the centre (subject matter department) Statisticians in the centre (subject matter department) allowed to modify data on specific consideration (apply the validation rules) allowed to modify data on specific consideration (apply the validation rules) ->Three groups of users – standard roles select – view all information update– enter&modify data (statisticians) admin – modify META information (IT experts)

Features of application framework Use META information Use META information documentation – measures, nomenclatures documentation – measures, nomenclatures code validation – nomenclature code, name code validation – nomenclature code, name meta driven function – access control: department/survey, user/survey meta driven function – access control: department/survey, user/survey Standard menu structure with the functions Standard menu structure with the functions flow control, data entry and validation, online and offline validation, reporting, management information, documents flow control, data entry and validation, online and offline validation, reporting, management information, documents Standard user interface Standard user interface font type & size, colours (input/display item) font type & size, colours (input/display item) navigation between tables/rows/items, using of mouse navigation between tables/rows/items, using of mouse

Validation rules Inside questionnaire Inside questionnaire a+b<=c, 01+02=03, 1T/05a+1T/05b=2T/01a Between questionnaires Between questionnaires same survey (previous periods), other surveys Information in META system - error message, error level Information in META system - error message, error level Statisticians (in the centre) define the categories Statisticians (in the centre) define the categories Categories (levels) Categories (levels) 1: warning– information 1: warning– information 2: caution – correct or user can accept with explanation 2: caution – correct or user can accept with explanation 3: error – must be corrected or accepted by superuser with explanation 3: error – must be corrected or accepted by superuser with explanation 4: critical – must correct immediately 4: critical – must correct immediately

Reusable building blocks Same functions – same solutions – reusable modules Code validation Code validation nomenclature code, enterprise identity code Standard first page Standard first page contact information, open text note from the enterprise Update response information in survey control system Update response information in survey control system Exception handling Exception handling error level: 2, 3 -> explanations are stored in DB Transform report format Transform report format reports in text or Excel (format transform module) Validation modules are generated Validation modules are generated predefined format (Excel) -> program code generator

Management information Survey status Survey status data entry or receiving e-questionnaires are allowed/permitted data entry or receiving e-questionnaires are allowed/permitted Processes Processes audit trail, who & when started the process audit trail, who & when started the process Questionnaires Questionnaires who & when entered/loaded, modified, deleted, controlled who & when entered/loaded, modified, deleted, controlled Errors Errors levels, status (accepted or not), who & when accepted, explanation text levels, status (accepted or not), who & when accepted, explanation text

Development rules Conventions, standards on object level Conventions, standards on object level compulsory tables and columns in the DB compulsory tables and columns in the DB folder structure in the file system folder structure in the file system strong name convention in the DB, in the file system strong name convention in the DB, in the file system

Control and quality Workflow control in the development Workflow control in the development controlled development phases, checkpoints, feedback controlled development phases, checkpoints, feedback Workflow control in production Workflow control in production more hundred users – harmonized and controlled activity more hundred users – harmonized and controlled activity audit trail, compulsory steps, alert messages audit trail, compulsory steps, alert messages Product control Product control evaluation reports, explanations and messages from the enterprise evaluation reports, explanations and messages from the enterprise

Advantages of standardisation ADEL - one common system for different questionnaires different questionnaires different years different yearsSolution unified data structure, conventions unified data structure, conventions common program elements, unified menu-structure common program elements, unified menu-structureResult unified user interface, unified workflow for statisticians – easy using unified user interface, unified workflow for statisticians – easy using saving development capacity saving development capacity unified quality and monitoring reports for all surveys from the common data tables unified quality and monitoring reports for all surveys from the common data tables

IT environment Developed with Oracle tools Developed with Oracle tools Development (on Windows XP) Development (on Windows XP) Designer Designer Developer Suite 10g (Form, Report-Writer, Sqlplus, PL/Sql) Developer Suite 10g (Form, Report-Writer, Sqlplus, PL/Sql) Running: WEB technology Running: WEB technology Client side: Windows XP, Sun Java 1.6 Client side: Windows XP, Sun Java 1.6 Application side: Red Hat Enterprise Linux, Oracle Application Server 10g Application side: Red Hat Enterprise Linux, Oracle Application Server 10g Database side: HP UNIX 11.23, RDBMS 10.2 Database side: HP UNIX 11.23, RDBMS 10.2