1. Impact Assessment Study of eGovernment Projects: Methodology and Some Preliminary Results
Subhash Bhatnagar
Indian Institute of Management, Ahmedabad
and
Advisor e-government, Information Solutions Group (Informatics Program)
World Bank, Washinton DC

2. Study Objectives
Review the status of impact assessment of eGovernment projects in developing countries
To develop and present an analytical framework to conduct analysis on the impact of e-delivery of services.
To study selected e-delivery projects, assess their impact, and identify key determinants of economic, organizational, and social impact.

3. Study Team Indian Institute of Management, Ahmedabad (IIMA)
Subhash Bhatnagar, Rama Rao, Nupur Singh, Ranjan Vaidya, Moushami Mandal
London School of Economics
Shirin Madon, Matthew Smith
eGov Practice Group
Deepak Bhatia, Jiro Tominaga, Soren Giggler
Reviewers
WBI, EAP, SAR, OECD, GICT,PREM, DEC
Sponsors
ISG,EAP,IIMA

4. Learning from Past Assessments
Variety of approaches have been used-client satisfaction surveys, expert opinion, ethnographic studies
Client satisfaction survey results can vary over time as bench mark changes—need for counterfactuals
Often studies have been done by agencies that may be seen as being interested in showing positive outcome
Lack of credibility of results-different studies of the same project show very different outcomes
Lack of rigor in sampling-results can not be easily generalized
Lack of rigor in controlling for external influence-need for counterfactuals ignored.
Lack of a standard methodology-making it difficult to compare projects
Hardly any projects do a benchmark survey

5. Critique of Existing Frameworks
Biased towards quantification of short term direct cost savings- quality of service, governance and wider impacts on society not studied.
Conceptual in nature-hardly any frameworks have been applied to assess impact of real projects
Variety in delivery models has not been recognized. Impact is a function of the delivery model and the nature of clients being served
Practical issues of paucity of data have not been taken into account-particularly in a developing country context where baseline surveys are not done and M&E systems are weak

6. Proposed Framework
Small budget exploratory study- focus on retrospective assessment of e-delivery systems(B2C and B2B)
Balanced approach between case study and quantitative analysis
Recognizes that some part of the value to different stakeholders can not be monetized
Understand how inputs lead to outputs and outcomes in different project contexts
A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of Bank projects in countries with various delivery models and paucity of data

7. Measurement Framework
Stakeholders
Key Dimension of Impact
Client
Economic (Direct & Indirect)
Governance (Corruption, Accountability, Transparency, Participation)
Quality of Service (Decency, Fairness, Convenience, etc.)
Over all satisfaction
Agency
(Including Partners in Implementation)
Performance on Key Non-economic Objectives
Process Improvements
Work life of employees
Society
Other Departments
Government as a Whole
Civil Society
Desirability of investments in eGovernment
Impact on vulnerable groups
Image of Government (Efficiency,
Corruption, Accountability, Transparency,
Participation, Responsiveness)
Impact on development Goals

8. Projects of e-delivery of Services
Issue of land titles in Karnataka (Bhoomi): 180 Kiosks, 18 million titles issued
Property registration in Karnataka (Kaveri): 230 offices
Computerized Treasury (Khajane): 240 locations
Property Registration in Andhra Pradesh: AP 400 offices. 5.7 million documents, 3.6 million encumbrance certificates, 2 million market valuation slips
eSeva center in Andhra Pradesh: 250 locations in 190 towns, Used by 3.5 million citizens,
eProcurement in Andhra Pradesh
Ahmedabad Municipal Corporation (AMC): 14 Civic Service Centers
Computerized Inter State Check Posts in Gujarat: 8 locations
eProcurement in Chile (Comprasnet)
Income Tax on-line in Chile

9. Methodology Used for Assessment
Selected 10 mature, wide scope and scale projects of e-delivery of services.
Collected data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system)
Survey instrument customized to each project, adapted in local language
Sampled 240 clients, 30 employees randomly from locations stratified by activity levels and remoteness
Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies.
Develop a case study-organizational context, process reform, change management.

10. Profile of AMC Respondents
Partial data of 139 (out of 240) citizens
40% graduates, 50% schooled, 10% barely literate
16% workers, 50% business, 10% white collar workers, 24% supervisor/management
Aware for 28 months, Average Income Rs 7000 pm
Urban population: 38% within 1-3 kms of a center, 25% 4-6 Kms, 20% < 1 Km
Payment of property tax, birth/death registration

11. Desirable Attributes in Service Delivery at AMC Centers
Less time and effort 48%
Less corruption %
Less cost to the citizen
Greater Transparency
No need for agents
Good complaint handling
Convenient time
Privacy and security
Waiting facilities
Helpful attitude of civil servants 10.8
Error free transactions 9.4
Fair treatment
Accountability
More predictable outcome 5.4
Clarity/simplicity of procedures 5.4

12. AMC: Impact on Client Costs
Manual
Computerized
Difference
Travel Costs (in Rupees)
22.2(17)
16.5(14)
5.7
Wait Time (in minutes)
27.3(21)
15.7(20)
11.6
Wage Loss (in Rupees)
119.8(135)
78.5(55)
41.3
Error Rate (% of responses)
2.16%
1.44%
0.72%
No of cases of Bribes (of 139) 5 1
Perception: not corrupt or low corruption (5, 4 on a 5 point scale)
40.2%
68.4%
28.2%
Document Preparation (in hours)
33.5(47)
20(30)
13.5
Stress levels: no stress, not much stress (5,4 on 5 point scale)
31%,31%
66%,23%
27%
Overall very satisfied (5 on a 5 point scale)
23%
69%
46%

13. Client Perceptions (Ratings on a 5 Point Scale)
Manual
HIGH LOW
Computerized
HIGH LOW
Time and effort
40.5
31.0
18.5 94 4 2
Less costs 36 28 85 12 3
Convenient schedule 51 29 20
78.5
16.5 5
Error free 50 32 18 80 14 6
Less corruption 46 25 81 16
Transparency 41 40 19 82
Accountability 30 88
Clarity of processes 83
Predictable outcome 48 26
Privacy and security 35 76

14. Attitude to eGovernment
Sample Statements
Strong
Agree
neutral
Disagree
Strong Disagree
More agencies should be computerized
65.5 26
6.5
1.5
0.7
Building of schools/dispensaries/roads is more beneficial than eGovernment 9
34.5 31 20
Computerization of Government department is a waste of resources
4.3
20.9
43.9
29.5
eGovernment benefits the rich and influential
5.0
26.6
38.8
28.1
Rural citizens benefit greatly from eGovernment
22.3 16
29.6
21.6
10.5
Government should make more investments in eGovernment
58.2 34
7.1 .7
Leads to impersonalization of public service
8.6 25
21.7
15.2
Improves the image of Government 50 39
2.4

15. Preliminary Observations
Overall Impact
Significant positive impact on cost of accessing service
Strong endorsement of eGovernment
Variability across different service centers of a project
Reduced corruption-outcome is fragile
Any type of system break down leads to corruption
Agents play a key role in promoting corruption
Private operators also exhibit rent seeking behavior given an opportunity Systematizing queues and appointments helps prevent break down
Small improvements in efficiency can trigger major positive change in perception about quality of governance.
Challenges
No established reporting standards for public agencies- In case of treasuries, the AG office has more information on outcome.
What is the bench mark for evaluation-past performance or potential?
Public agencies are wary of evaluation-difficult to gather data