Scanning Technology and Its Application in Ethiopia Yakob Mudesir Deputy Director General Central Statistical Agency of Ethiopia Web: October 2009 Kampala, Uganda

Outline: Background Scanning Technology Requirements for Scanning The Ethiopian Experience

 Population and Housing Census process is the largest data capturing exercise that a country can undertake.  It involves capturing of millions of forms  The Central Statistics Agency (CSA) started using old techniques like Punched Card Reader as early 1960’s.  Two Population and Housing Censuses have so far been conducted in Ethiopia using the traditional method of data capturing.  The first Population and Housing Census was carried out in Background

 During the 1984 Census:  Data capture was done on manual keyboard based entry using mainframe computer  FORMSPEC data entry system was used  It took more than 2 years to capture the data for about 42 million people.  In the case of the 1994 Census:  Data capture was again done on manual keyboard entry basis using PC’s  CENTRY data entry system (IMPS) was used Background

 It took about 18 months to capture the data for the population of about 53 million.  The entry work was done on 2-shift basis  About 180 data entry clerks were involved  Around 90 Pc’s were used Background

 Time consuming  Does not allow the availability of timely data  The data will be weaker in representing the current or existing situation  Subject to additional non-sampling errors  Human error due to manual keying  Due to the volume of the data, a 100% verification as in the case of sample surveys, is difficult. Some Limitations of the Keyboard Manual Entry Method

 Involves a great deal of human resource management.  Large number of data entry operators and equipment required Some Limitations of the Keyboard Manual Entry Method

The need to have the census result right on time and the limitations discussed above forced the statistical offices to look for other alternatives.. specially related to large volume of data - The Emerging of Scanning Technology The need for alternative solution

 The Scanning Technology in general implements two basic techniques  Mark recognition, like the Optical Mark Reader (OMR)  Character recognition, like the Optical Character Recognition (OCR) and the Intelligent Character Recognition (ICR) The Scanning Technology

 OMR is the recognition of shaded marks (blobs) on the forms  The positioning of these blobs on a form determines the alphanumeric characters they represent  The character recognition is the recognition of alphanumeric characters on forms and they are of 2 types:  OCR which is the recognition of machine printed characters and..  ICR which refers to the capture of hand- printed characters from a form The Scanning Technology

 Significant decrease in time required to capture the data  This helps to get timely data  Users’ need satisfied (policy makers, planners, researchers, etc.)  Reduces the non sampling error  No need to worry to store millions of forms for possible future references  Scanning captures the whole content of a questionnaire in an electronic image format Major Benefits of the Scanning Technology

EXPORTEXPORT Scanning Recognition & Extraction The Process involved in Scanning

- Proper planning - On the DESIGN of the questionnaire - On PRINTING the questionnaire - On RECORDING answer - On Questionnaire HANDLING - On securing ADEQUATE SPACE for questionnaire movement Why? In order to minimize the rejection rate and increase the Recognition Rate Requirements for Effective Scanning

 Proper training  Both on Hardware and Software  This helps to “ own” the technology  Being able to use the technology after the departure of the trainers / technical advisors  A well organized space for forms and data flow is required Requirements for Effective Scanning

- A Reliable Network System Requirements for Effective Scanning

Data Processing Center Warehouse Registering the EA Received from the Field Receiving Receiving the Questionnaires Registering the EA for Scanning Waiting Room Scanning Room Validation Room Processing Room Store A WELL STRUCTURED SPACE FOR FILE FLOW Requirements for Effective Scanning

File recorder: Records the ID of the outgoing questionnaires Filing Box handler Checks the village code on the box is the same as on the questionnaire Machine room box handler Checks the number, orientation and damage to questionnaires Guillotine Machine operator Cuts the binding edge off all questionnaires in the village in one operation and places them on the scanner Scanner technician Ensures smooth running of the machine and assists with the paper handling BO X Scanner Paper Handler Responsible for checking paper throughput through the scanner Computer operators (x2) Responsible detecting errors in the scanning process Stitching machine operator lifts paper off scanner and stitches in the bottom left corner Machine room box handler returns box to the File recorder Filing Box handler Returns the questioanires to the shelves File recorder: Records the ID No of the incomming questionnaires. Loos e Pap er Principal Supervisor Senior Supervisor Scanner Supervisor Proper File Management Requirements for Effective Scanning

 Proper file management and care  Checking batch (EA) IDs and orientation of forms  Proper recording of the in-coming and out- going questionnaires  Ensuring the EA code on each box is the same as the one on the questionnaires  Close attention in detecting errors in the scanning process is required Requirements for Effective Scanning

The Ethiopian Experience

 Study tour made in two African countries  Tanzania  To learn from their successes  Data capture of the 2002 Census of Tanzania was done in about 26 days  General report tables were produced within 3 months from the start of the scanning Experience Sharing

 Ghana  To learn from their difficulties  Data capture of the 2000 Census took about 6 months - ( forms from 29,000 EAs)  3 Scanners were used (Kodak, Fujitsu) > The larger scanner was Kodak 500D > Speed: About 500 forms/min  Power failure was one of the major problems > Loss of some data occurred as a result > A large generator was installed to minimize the effect of the frequent power cut Experience Sharing

Identification of the Technology  For scanning of the 2007 Census the Optical Mark Reader (OMR) technique was used  Scanning Technology to be used PhotoScribe Series PS900 Scanners DRS Scanning Technology product

DRS Photo Scribe Series PS900  High speed Imaging Mark Reader  Windows XP professional  Network connectivity  CD R/WR drive  A TFT monitor, Keyboard, mouse  Speed: up to 8,500 forms / hour Identification of the Technology

Design and Printing of Forms  Types of the 2007 Census forms  Short questionnaires  Long questionnaires  Household Listing Forms  Summary Forms  Community Level Forms  Batch Header Form – Scanned to create EA database

Long Questionnaire Design and Printing of Forms

Control Database Form Summary Form Design and Printing of Forms

 Data from the Pilot Census successfully scanned (OMR), key-corrected, exported to text format, tabulated and tested.  One scanner (PS 900 Photo Scribe) was used to capture the pilot data  Technical experts from the DRS company assisted in capturing, validating and exporting the pilot data Pilot

 Hardware and Software training  16 professionals trained  The training in general took about 7 working days  SOSKITW for Windows:- a DRS software package for scanning was used  Components of the SOSKITW Software :  SOSGen : - used to generate scanning decodes for completed OMR forms (How marks on forms are interpreted and stored)  SOSInp : - used to scan, validate and export scanned data. Training

 Equipment purchased and installed  10 additional PS900 iM2 DRS Scanners  16 high capacity PC’s for key-correction  Census data processing work plan prepared  Recruitment of temporary staff  Staff training (scanning technology, CSPro)  Retrieval and organization of completed forms  Scanning and validation  Computer editing and tabulation (For each activity: duration and responsible body are indicated) Preparatory Activities

 About 33 teams for registering and organizing forms are organized 3 persons assigned per team  Census data processing teams organized  Batch header database group  Scanning and validation team  Shift supervisors  Two senior programmers responsible for the overall scanning process  Other sub-professional staff assigned  4 batch header scanning technicians  16 data validation workers Preparatory Activities

–The scanning room organized – An air conditioner for the scanning room installed – A high capacity automatic generator installed to ensure uninterrupted power supply Preparatory Activities

 Organized forms taken from store to the waiting room  Batch header information printed and associated with its respective EA box  The existence of each EA verified  Checked EAs sent to the scanning room  Scanned forms are finally sent back to the stores  Scanned data are validated / key-corrected The Scanning Process

The actual scanning started mid July and the scanning work has been completed in November 2007 The Scanning Process

 Scanned, key-corrected and exported data  Batch Edit program based on edit specs provided by subject matter specialists developed and run on the data.  The software to be used in editing the data will be the Census and Survey Processing System (CSPro) Data Cleaning / Computer Editing

Owning the Technology Two Professionals have been trained in England for two weeks on scanned document processing using DocXP in close collaboration with DRS Printing of the Questionnaire locally Questionnaire design Our Professionals are working hard to process the upcoming welfare Monitoring Survey using scanning technology Future Plan

THANK YOU