Impact Assessment Study of eGovernment Projects: Methodology and Some Preliminary Results Subhash Bhatnagar Indian Institute of Management, Ahmedabad (subhash@iimahd.ernet.in) and Advisor e-government, Information Solutions Group (Informatics Program) World Bank, Washinton DC sbhatnagar@worldbank.org

Study Objectives Review the status of impact assessment of eGovernment projects in developing countries To develop and present an analytical framework to conduct analysis on the impact of e-delivery of services. To study selected e-delivery projects, assess their impact, and identify key determinants of economic, organizational, and social impact.

Study Team Indian Institute of Management, Ahmedabad (IIMA) Subhash Bhatnagar, Rama Rao, Nupur Singh, Ranjan Vaidya, Moushami Mandal London School of Economics Shirin Madon, Matthew Smith eGov Practice Group Deepak Bhatia, Jiro Tominaga, Soren Giggler Reviewers WBI, EAP, SAR, OECD, GICT,PREM, DEC Sponsors ISG,EAP,IIMA

Learning from Past Assessments Variety of approaches have been used-client satisfaction surveys, expert opinion, ethnographic studies Client satisfaction survey results can vary over time as bench mark changes—need for counterfactuals Often studies have been done by agencies that may be seen as being interested in showing positive outcome Lack of credibility of results-different studies of the same project show very different outcomes Lack of rigor in sampling-results can not be easily generalized Lack of rigor in controlling for external influence-need for counterfactuals ignored. Lack of a standard methodology-making it difficult to compare projects Hardly any projects do a benchmark survey

Critique of Existing Frameworks Biased towards quantification of short term direct cost savings- quality of service, governance and wider impacts on society not studied. Conceptual in nature-hardly any frameworks have been applied to assess impact of real projects Variety in delivery models has not been recognized. Impact is a function of the delivery model and the nature of clients being served Practical issues of paucity of data have not been taken into account-particularly in a developing country context where baseline surveys are not done and M&E systems are weak

Proposed Framework Small budget exploratory study- focus on retrospective assessment of e-delivery systems(B2C and B2B) Balanced approach between case study and quantitative analysis Recognizes that some part of the value to different stakeholders can not be monetized Understand how inputs lead to outputs and outcomes in different project contexts A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of Bank projects in countries with various delivery models and paucity of data

Measurement Framework Stakeholders Key Dimension of Impact Client Economic (Direct & Indirect) Governance (Corruption, Accountability, Transparency, Participation) Quality of Service (Decency, Fairness, Convenience, etc.) Over all satisfaction Agency (Including Partners in Implementation) Performance on Key Non-economic Objectives Process Improvements Work life of employees Society Other Departments Government as a Whole Civil Society Desirability of investments in eGovernment Impact on vulnerable groups Image of Government (Efficiency, Corruption, Accountability, Transparency, Participation, Responsiveness) Impact on development Goals

Projects of e-delivery of Services Issue of land titles in Karnataka (Bhoomi): 180 Kiosks, 18 million titles issued Property registration in Karnataka (Kaveri): 230 offices Computerized Treasury (Khajane): 240 locations Property Registration in Andhra Pradesh: AP 400 offices. 5.7 million documents, 3.6 million encumbrance certificates, 2 million market valuation slips eSeva center in Andhra Pradesh: 250 locations in 190 towns, Used by 3.5 million citizens, eProcurement in Andhra Pradesh Ahmedabad Municipal Corporation (AMC): 14 Civic Service Centers Computerized Inter State Check Posts in Gujarat: 8 locations eProcurement in Chile (Comprasnet) Income Tax on-line in Chile

Methodology Used for Assessment Selected 10 mature, wide scope and scale projects of e-delivery of services. Collected data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system) Survey instrument customized to each project, adapted in local language Sampled 240 clients, 30 employees randomly from locations stratified by activity levels and remoteness Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies. Develop a case study-organizational context, process reform, change management.

Profile of AMC Respondents Partial data of 139 (out of 240) citizens 40% graduates, 50% schooled, 10% barely literate 16% workers, 50% business, 10% white collar workers, 24% supervisor/management Aware for 28 months, Average Income Rs 7000 pm Urban population: 38% within 1-3 kms of a center, 25% 4-6 Kms, 20% < 1 Km Payment of property tax, birth/death registration

Desirable Attributes in Service Delivery at AMC Centers Less time and effort 48% Less corruption 41.7% Less cost to the citizen 29.5 Greater Transparency 27.3 No need for agents 20.1 Good complaint handling 19.4 Convenient time 16.5 Privacy and security 15.1 Waiting facilities 12.2 Helpful attitude of civil servants 10.8 Error free transactions 9.4 Fair treatment 8.6 Accountability 7.9 More predictable outcome 5.4 Clarity/simplicity of procedures 5.4

AMC: Impact on Client Costs Manual Computerized Difference Travel Costs (in Rupees) 22.2(17) 16.5(14) 5.7 Wait Time (in minutes) 27.3(21) 15.7(20) 11.6 Wage Loss (in Rupees) 119.8(135) 78.5(55) 41.3 Error Rate (% of responses) 2.16% 1.44% 0.72% No of cases of Bribes (of 139) 5 1 Perception: not corrupt or low corruption (5, 4 on a 5 point scale) 40.2% 68.4% 28.2% Document Preparation (in hours) 33.5(47) 20(30) 13.5 Stress levels: no stress, not much stress (5,4 on 5 point scale) 31%,31% 66%,23% 27% Overall very satisfied (5 on a 5 point scale) 23% 69% 46%

Client Perceptions (Ratings on a 5 Point Scale) Manual HIGH LOW Computerized HIGH LOW Time and effort 40.5 31.0 18.5 94 4 2 Less costs 36 28 85 12 3 Convenient schedule 51 29 20 78.5 16.5 5 Error free 50 32 18 80 14 6 Less corruption 46 25 81 16 Transparency 41 40 19 82 Accountability 30 88 Clarity of processes 83 Predictable outcome 48 26 Privacy and security 35 76

Attitude to eGovernment Sample Statements Strong Agree neutral Disagree Strong Disagree More agencies should be computerized 65.5 26 6.5 1.5 0.7 Building of schools/dispensaries/roads is more beneficial than eGovernment 9 34.5 31 20 Computerization of Government department is a waste of resources 4.3 20.9 43.9 29.5 eGovernment benefits the rich and influential 5.0 26.6 38.8 28.1 Rural citizens benefit greatly from eGovernment 22.3 16 29.6 21.6 10.5 Government should make more investments in eGovernment 58.2 34 7.1 .7 Leads to impersonalization of public service 8.6 25 21.7 15.2 Improves the image of Government 50 39 2.4

Preliminary Observations Overall Impact Significant positive impact on cost of accessing service Strong endorsement of eGovernment Variability across different service centers of a project Reduced corruption-outcome is fragile Any type of system break down leads to corruption Agents play a key role in promoting corruption Private operators also exhibit rent seeking behavior given an opportunity Systematizing queues and appointments helps prevent break down Small improvements in efficiency can trigger major positive change in perception about quality of governance. Challenges No established reporting standards for public agencies- In case of treasuries, the AG office has more information on outcome. What is the bench mark for evaluation-past performance or potential? Public agencies are wary of evaluation-difficult to gather data