1. Impact Assessment Monitoring & Evaluation for e-Government Projects
Subhash Bhatnagar
As part of the Capacity Building Workshop under the Joint Economic Research Program (JERP)

2. This session will focus on the need for assessing impact of e-government projects and will described a methodology of how such an assessment can be carried out. Results of an impact assessment study sponsored b the World Bank will be discussed in detail illustrating the methodology and the value that could be derived from such assessments. Some of the pitfalls that should be avoided in making assessment will be described.

3. Presentation Structure
Why assess impact?
Learning from past work on assessment
Proposed Methodology
Results from a Bank Study of 8 projects
Study objectives
Projects covered in the study
Analysis of Results
Are investments in eGovernment worthwhile?
Lessons for assessment work

4. Why Impact Assessment?
To ensure that funds deployed in eGovernment provide commensurate value.
To create a bench mark for future projects to target
To identify successful projects for replication and scaling up
To sharpen goals and targeted benefits for each project under implementation
To make mid course correction for projects under implementation
To learn key determinants of economic, organizational, and social impact from successful and failed projects

5. Evaluation of Impact: Key Issues
Macro versus Micro Approach- unit of analysis
Assessment from whose perspective?
Dimensions on which impact can be assessed for different stakeholders
Can all costs and benefits be monetized?
How to isolate the effect of ICT use from different interventions ?
Degree of quantification versus qualitative assessment
Measurement issues: sampling, questionnaire design, analysis of internal data, triangulation

6. Learning from Past Assessments
Variety of approaches have been used-client satisfaction surveys, expert opinion, ethnographic studies
Client satisfaction survey results can vary over time as bench mark changes - need for counterfactuals
Often studies have been done by agencies that may be seen as being interested in showing positive outcome
Lack of credibility of results-different studies of the same project show very different outcomes
Lack of rigor in sampling-results can not be easily generalized
Lack of rigor in controlling for external influence-need for counterfactuals ignored.
Lack of a standard methodology-making it difficult to compare projects
Hardly any projects do a benchmark survey

7. Critique of Existing Frameworks
Biased towards quantification of short term direct cost savings- quality of service, governance and wider impacts on society not studied.
Conceptual in nature-hardly any frameworks have been applied to assess impact of real projects
Variety in delivery models has not been recognized. Impact is a function of the delivery model and the nature of clients being served
Practical issues of paucity of data have not been taken into account-particularly in a developing country context where baseline surveys are not done and M&E systems are weak

8. Measurement Framework
Stakeholders
Key Dimension of Impact
Client
Economic (Direct & Indirect)
Governance (Corruption, Accountability, Transparency, Participation)
Quality of Service (Decency, Fairness, Convenience, etc.)
Over all satisfaction
Agency
(Including Partners in Implementation)
Performance on Key Non-economic Objectives
Process Improvements
Work life of employees
Society
Other Departments
Government as a Whole
Civil Society
Desirability of investments in e-Government
Impact on vulnerable groups
Image of Government (Efficiency, Corruption, Accountability, Transparency, Participation, Responsiveness)
Impact on development Goals

9. Proposed Framework
Focuses on retrospective assessment of e-delivery systems(B2C and B2B)
Balanced approach between case study and quantitative analysis
Recognizes that some part of the value to different stakeholders can not be monetized
Understand how inputs lead to outputs and outcomes in different project contexts
A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of projects in countries with various delivery models and paucity of data

10. Methodology for Assessment
Select mature, wide scope and scale projects of e-delivery of services.
Collect data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system)
Customize survey instrument to each project, adapt in local language
Data can be collected through Internet survey, face to face interviews and focus groups
Use professional market research agencies with trained investigators for face to face int
Determine sample frame and size so that results can be extrapolated to the entire population (often 300 clients may be sufficient). Select respondents randomly from locations stratified by activity levels and remoteness
Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies.
Develop a case study-organizational context, process reform, change management.

11. A study sponsored by World Bank Done by Indıan Instıtute of Management Ahmedabad and London School of Economıcs Preliminary Results from Projects in India

12. Study Team Study Coordinator: Subhash Bhatnagar
Indian Institute of Management, Ahmedabad (IIMA)
Subhash Bhatnagar, Rama Rao, Nupur Singh, Ranjan Vaidya, Mousumi Mandal
London School of Economics
Shirin Madon, Matthew Smith
ISG e-Gov Practice Group
Deepak Bhatia, Jiro Tominaga
Sponsors
World Bank,IIMA, Department of IT

13. Projects of e-delivery of Services
Issue of land titles in Karnataka (Bhoomi): 180 Kiosks, Launched February 2001 (2-01)
Property registration in Karnataka (Kaveri): 230 offices (3-03)
Computerized Treasury (Khajane): 240 locations (11-02)
Property Registration in Andhra Pradesh: AP 400 offices. (11-98)
eSeva center in Andhra Pradesh: 250 locations in 190 towns, Used monthly by 3.5 million citizens (8-01)
e-Procurement in Andhra Pradesh (1-03)
Ahmedabad Municipal Corporation (AMC): 16 Civic Service Centers (9-02)
Inter State Check Posts in Gujarat: 10 locations (3-2000)
e-Procurement in Chile (Comprasnet)
Income Tax on-line in Chile

14. Dimensions to be Studied to Evaluate Impact
Project context: basic information on the project and its context
Inputs (technology, human capital, financial resources);
Process outcome (reengineered processes, shortened cycle time, improved access to data and analysis, flexibility in reports);
Customer results (service coverage, timeliness and responsiveness, service quality and convenience of access);
Agency outcomes (transparency and accountability, less corruption, administrative efficiency, revenue growth and cost reduction) and
Strategic outcomes (economic growth, poverty reduction and achievement of MDGs).
Organizational processes: institutional arrangements, organizational structure, and other reform initiatives of the Government that might have influenced the outcome for the ICT project.

15. Profile of Respondents

16. Improvement Over Manual System
AMC
CARD
Check Post
E-Proc
E-Seva
Kaveri
Bhoomi
Total Travel Cost per transactıon (Rs.)
21.07
67.71
7.40
89.22
0.15
Number of trips
0.65
1.38
5.16
0.28
1.18
0.47
Wage Loss (Rs.)
36.84
28.46
15.63
120.55
(39.22)
Waiting Time (Minutes)
14.69
97.00
8.94
114.95
18.50
61.81
33.97
Governance Quality - 5 point scale
1.08
1.01
0.25
Percentage paying bribes
2.51
4.31
6.25
11.77
0.40
12.71
18.83
Service Quality- 5 point scale
0.52
0.58
0.24
0.76
0.27
0.85
Error Rate
0.42
0.86
1.58
3.80
0.03
Preference for Computerization (%)
97.49
96.98
91.25
83.71
96.84
98.31
79.34

17. Savings in Cost to Customers Estimates for entire client population
Projects
Million Transactions
Travel Cost Saving (Rs. Million)
Wage Loss
(Rs. Million)
Waiting Time (Hours)
Bribes (Rs. Million)
Additional Revenue (Rs. Million)
Bhoomi
RTC
MUT
(73.96)
(0.086)
66.40
KAVERI
1.0277
91.69
123.89
1.059
(49.40)
CARD
1.0295
69.71
29.30
1.665
(95.99)
e-Seva
37.20
275.45
581.40
11.468 NA
e-Procurement
.0264
90.73
0.0507
3.38
AMC
0.6171
13.43
22.70
0.151
0.15
Check Post
2.444
270.37

18. Projects: Descending Order Of Improvement in Composite Scores on a 5 point scale
Manual
Computer
Difference Average
Average
S.D.
BHOOMI
2.86
0.86
4.46
0.51
1.60
e-SEVA
3.39
0.65
4.66
0.39
1.27
e-PROCUREMENT
3.22
0.58
4.26
1.03
CHECK POST
3.48
0.79
4.32
0.59
0.84
AMC
3.37
0.61
4.12
0.90
0.75
KAVERI
3.35
3.90
0.74
0.55
CARD
3.78
0.49
3.93
0.38
0.15

19. Descending Order Of Post Computerization Composite Score on a 5 point scale
Project
Manual
Computer
Difference Average
Average
S.D.
e-Seva
3.39
0.65
4.66
0.39
1.27
Bhoomi
2.86
0.86
4.46
0.51
1.60
Check Post
3.48
0.79
4.32
0.59
0.84
e-Procurement
3.22
0.58
4.26
1.03
AMC
3.37
0.61
4.12
0.90
0.75
CARD
3.78
0.49
3.93
0.38
0.15
KAVERI
3.35
3.90
0.74
0.55

20. Client Perception (Rating on 5 Point Scale in AMC)
Difference
weighted Scores
Computerized
Frequency Distribution
Manual
High
Low
Less costs
0.98
78.2
10.5
11.3
36.4
36.8
26.8
Good Waiting Facilities
0.97
79.5
13.0
7.5
41.4
31.8
Time and effort
0.90
85.8
3.8
46.4
37.7
15.9
Complaint Handling
0.77
75.7
43.1
32.6
24.3
Greater Transparency
0.68
16.7
46.0
36.0
18.0
Fair Treatment
0.65
77.8
15.1
7.1
51.0
33.5
15.5
No need for Agents
0.52
71.5
23.4
5.0
57.3
25.9
Equal Opportunity
0.47
74.5
10.0
55.2
29.3

21. Top Four Attributes Desired in the Application
AMC
Less time and effort required
Less corruption
Greater transparency
Good complaint handling system
CARD
Less waiting time
Fair Treatment
e-Procurement
Easy access
Equal opportunity to all
Transparent system of tender valuation
e-Seva
Convenient time schedule
Check Post
No Delay in transactions
Error Free
Payment receipts
Error-free transactions
Bhoomi
Error free transaction
No delay in transaction
KAVERI
Less Corruption

22. Impact on Agency AMC Civic Center CARD e-Seva Bhoomi KAVERI Check post
eProcurement
Khajane
Total Project Investment (Rs. million)
250.00
300.00
537.00
330.00
185.00
3.50
50.4
338.00
Operating Expenses
168.9
52.7
24.3
64.9
Annual Transactions (million)
0.71
1.03
37.20
2.84
2.47
16.73
0.03
15.69
Clients Served (million)
0.29
0.33
1.89
1.67
1.33
6.12
0.00
Tax Revenue in for Computerized (Rs. million)
1974.2
17282
19245
3109.4
Tax Revenue in Last Year of Manual (Rs. million)
42.05
9033
702.68
Growth Rate in Tax Revenue for Computerized
31.95
50.17
17.00
15.10
Transaction Fees in Computerized (Rs. million)
53.32
1130.8
203.59
274.19
2626.9
Transaction Fees in Last Year of Manual (Rs. million)
2.53
Growth Rate in Transaction Fees for Computerized
12.86
50.90
83.51
22.76
16.71
113.06
8.06

23. Agency: Growth of Tax and Transaction Fee

24. Economic Viability of Projects Agency Perspective
Yearly Operating Expense per Transaction
Investment per Cumulative Transactions for 4 years
AMC Civic Center
109.42
CARD
95.94
e-Seva
4.56
6.57
Bhoomi
18.54
9.48
KAVERI
Checkpost
2.76
eProcurement
918.85
43.70
Khajane
4.14
5.48

25. Attitude to e-Government

26. Preliminary Observations
Overall Impact
Significant positive impact on cost of accessing service
Variability across different service centers of a project
Strong endorsement of e-Government but indirect preference for private participation
Reduced corruption-outcome is mixed and can be fragile
Any type of system break down leads to corruption
Agents play a key role in promoting corruption
Private operators also exhibit rent seeking behavior given an opportunity
Systematizing queues by appointments helps prevent break down
Small improvements in efficiency can trigger major positive change in perception about quality of governance.
Challenges
No established reporting standards for public agencies- In case of treasuries, the AG office has more information on outcome.
What is the bench mark for evaluation-improvement over manual system, rating of computerized system (moving target), or potential?
Measuring what we purport to measure: design of questions, training, pre test, field checks, triangulation
Public agencies are wary of evaluation-difficult to gather data

27. Questionnaire Design and Survey
Design of analytical reports prior to survey. Often key variables can be missed if the nature of analysis in not thought through prior to the study.
Pre code as many items in the questionnaire as possible.
Consistent coding for scales -representing high versus low or positive versus negative perceptions.
Differently worded questions to measure some key items/ perceptions.
Wording of questions should be appropriate to skill level of interviewer and educational level of respondent.
Local level translation using colloquial terms.
Feedback from pre-testing of questionnaire should be discussed between study team and investigators. The feedback may include: the length of questionnaire, interpretation of each question and degree of difficulty in collecting sensitive data.
Quality of supervision by MR agency is often much worse than specified in the proposal. Assessing the quality of investigators is a good idea.
Involvement of study team during the training of investigators.
Physical supervision by study team of the survey process is a good idea, even if it is done selectively

28. Establishing Data Validity
Check extreme values in data files for each item and unacceptable values for coded items.
Cross check the data recorded for extreme values in the questionnaire.
Check for abnormally high values of Standard Deviation.
Even though a code is provided for missing values, there can be confusion in missing values and a legitimate value of zero.
Look for logical connections between variables such as travel mode and travel time; bribe paid and corruption.
Poor data quality can often be traced to specific investigators or locations.
Random check for data entry problems by comparing data from questionnaires with print out of data files.
Complete data validity checks before embarking on analysis