Deepak T. Bhatia Measuring the impact of Egovernment Deepak T. Bhatia Sectoral and E-government Applications Practice Lead Global ICT department (GICT) The World Bank

Deepak T. Bhatia It is no longer acceptable for technology to be utilized simply for technology’s sake. Individual e-government projects need to demonstrate their contribution to overall government objectives – with the focus being on results and impact.

Deepak T. Bhatia3 Why e-Government? Battling bureaucracy Improving Efficiency Better service delivery Strengthening governance and accountability

Deepak T. Bhatia4 About 150 contain ICT components (out of 250 new WB projects each year). In the World Bank the Number of Projects with ICT is growing 57% of active projects have ICT components (this is 930 projects) ICT components’ combined volume is $6.2 billion

Deepak T. Bhatia5 Analysis of e-Government Impact World Bank commissioned study to explore impact of e- government on citizens, in collaboration with the Indian Institute of Management, Ahmedabad, India and the London School of Economics Effort to go beyond anecdotal assessments commonly available Sample of 5 mature, wide scope/scale projects selected for analysis from India –Chosen to represent different stages of overall IT readiness and evolution

Deepak T. Bhatia Learning from Past Assessments Variety of approaches have been used for assessments -client satisfaction surveys, expert opinion Often studies have been done by agencies that may be seen as being interested in showing positive outcome Lack of credibility of results-different studies of the same project show very different outcomes Lack of rigor in sampling-results can not be easily generalized Lack of rigor in controlling for external influence-need for counterfactuals ignored. Lack of a standard methodology-making it difficult to compare projects

Deepak T. Bhatia7 Dimensions of measurement - Stakeholders Impact on Clients: All measurements on the basis of a sample of clients for each major service availed by the client. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service. In cases where alternate (non electronic modes) are currently being used by the same set of users in similar contexts elsewhere, measurement would be recorded for such usage Impact on Agencies (Including Partners in Implementation ): All measurements on the basis of data collected from agency records, a sample of employees or a group of managers. Measurements would be done for electronic delivery of services as well as for the earlier mode of delivery of the same service Impact on Society : Some of the questions to be framed as degree of improvements resulting from the implementation of a specific project

Deepak T. Bhatia Measurement Framework - Dimensions StakeholdersKey Dimension of Impact ClientEconomic (Direct & Indirect) Governance (Corruption, Accountability, Transparency, Participation) Quality of Service (Decency, Fairness, Convenience, etc.) Over all satisfaction Agency (Including Partners in Implementation) Economic (Direct & Indirect) Governance (Corruption, Accountability, Transparency, Participation) Performance on Key Non-economic Objectives Process Improvements Work life of employees Society Other Departments Government as a Whole Civil Society Desirability of investments in e-Government Impact on vulnerable groups Image of Government (Efficiency, Corruption, Accountability, Transparency, Participation, Responsiveness) Impact on development Goals

Deepak T. Bhatia Proposed Framework Focuses on retrospective assessment of e-delivery systems(B2C and B2B) Balanced approach between case study and quantitative analysis Recognizes that some part of the value to different stakeholders can not be monetized Understand how inputs lead to outputs and outcomes in different project contexts A practical methodology that can be used for designing bench mark surveys, M&E systems and prospective evaluation of projects in countries with various delivery models and paucity of data

Deepak T. Bhatia Methodology for Assessment Select mature implementations, of e-delivery of services. Collect data through structured survey from clients, employees, supervisors using counterfactuals ( for old non computerized delivery and new e-delivery system) Customize survey instrument to each project, adapt in local language Data can be collected through Internet survey, face to face interviews and focus groups Determine sample frame and size so that results can be extrapolated to the entire population (often 300 clients may be sufficient). Collect data on investments, operating costs, activity levels, revenues, employee strength from agencies

Deepak T. Bhatia11 Projects assessed Bhoomi project delivering two online services: issuance of records of rights, tenancy, and crop inspection register (RTC) and filing of requests for mutation (changes in land ownership) for affecting changes in land records. Karnataka Valuation and E-Registration (KAVERI) project delivering three key services: registration of property sale and purchase deeds, issuance of nonencumbrance certificates, and issuance of copies of previously registered deeds. Computer-aided Administration of Registration Department (CARD) same three online services provided by KAVERI. eSeva. 2002, These one-stop service centers now deliver 135 services from central, state, and local governments and public utilities, and are used by 3.1 million citizens a month Ahmedabad Municipal Corporation (AMC) Civic Centers project. 2002, deliver three important services: annual payment of property taxes, issuance of birth and death certificates, and issuance of shop licenses.

Deepak T. Bhatia12 Caveat largely relies on the survey of stakeholder perceptions of public services before and after these projects. Thus not an impact evaluation of investments in narrowly defined ICT, which typically includes hardware, software, and communication equipment. Rather, a review of the outcomes of packaged efforts involved in the transition from manual services to those using ICT as a key component.

Deepak T. Bhatia13 Impact: Computerized service delivery is preferable to manual

Deepak T. Bhatia14 User reports of changes in travel costs, by project (number of trips * distance travelled) Rupees per service Bhoomi costs went up due to change in location of service centers from villages to “taluka” headquarters

Deepak T. Bhatia15 User reports of reductions in waiting times, by project Minutes decreased in all five projects. With manual systems, users often had to wait in long lines to receive services. Automation has increasedthe efficiency of document processing.

Deepak T. Bhatia16 User perception of improvement in service quality Rating on elements such as service delivery location, accessibility, convenience, cost, transparency, and service orientation

Deepak T. Bhatia17 ProjectFeatures BhoomiError-free transactionsReduced delays in transactions Shorter waiting timesFewer visits KAVERILess corruptionIncreased transparency Error-free transactionsShorter waiting times CARDIncreased convenienceShorter waiting timesLess corruptionFair treatment eSevaIncreased convenienceShorter waiting timesMore convenient time schedules Fair treatment AMCIncreased convenienceLess corruptionGreater transparency Good complaint handling system Users’ top four desired features of services, by project

GoalsExamples of performance indicators Key features of the enabling environment Outcomes  Increased efficiency  Increased transparency and accountability  Higher-quality public services  Better access to services  Financial and time savings in government activities  Public perceptions, such as user satisfaction and score cards  Financial and time savings for citizens  Increased public service timeliness and responsiveness  Reduced errors  Financial saving per transaction  Overall e-government strategies  Political and popular support for cross agency coordination and public sector reform  Telecommunications infrastructure and cost structures for increasing ICT access  Supportive legal and regulatory frameworks  Balances with competing priorities (such as roads and education)  Macroeconomic changes Outputs  Reengineered processes  New ICT systems  Increased service coverage  Comparisons of old and new business processes  Technical reviews of IT infrastructure, applications, and performance  Variety of available services  IT support capacity  Service training Proposed conceptual framework

Deepak T. Bhatia19 Additional conclusions Out of the three projects where significant corruption was reported in the manual system, one project was able to eliminate corruption through computerization Incremental operating costs for agencies per transaction are less than the reduction in the direct cost of accessing the service reported by the clients. Negative aspect is the observed variability of impacts across different delivery centers in a project Agencies: ability to cope with growth in transactions was enhanced Citizens strongly supported the idea that more agencies need to be computerized, but not necessarily through government investments

Deepak T. Bhatia20 Conclusions Undertake ex-post evaluation of successful ongoing projects. Can be used for for ex-ante evaluation, with following assumptions –extent and adoption rate of user demand need to be estimated –Project designs and investment levels must be commensurate with anticipated benefits to users. –quality of the enabling environment should be assessed to gauge the expected impacts of planned projects –Detailed surveys at the initial stages of project planning, would help establish a baseline.

Deepak T. Bhatia21 Lessons Learned IT cannot be a goal in itself - projects must have a direct link to process reform E-Government is a long term endeavor: not a silver bullet, not for immediate gains ‘E’ stands for ‘Everything-but-IT’: institutional change, political support, capacity development.. technology is the easy part Ultimately can generate significant cost and efficiency gains But requires…

Deepak T. Bhatia22 Focusing on results

Deepak T. Bhatia23 Credits Impact Study team Subhash Bhatnagar, IIM, Ahmedabad, India (Primary Author) Jiro Tominaga, WB Deepak Bhatia, WB Shirin Madon, London School of Economics Presentation research Asheeta Bhavnani, WB Mikhail Bunchuk, WB Reviewers David L. Cieslikowski Christine Zhen-Wei Qiang

Deepak T. Bhatia24 Annex

Deepak T. Bhatia25 How have governments been measuring results of e- government investments? France – Mareva - A Method of Analysis and Value Enhancement Germany -WiBe - Economic Efficiency Assessment Methodology USA – Performance Reference Model Framework Australia: Demand and Value Assessment Methodology All these government methodologies broadly focus on the same dimensions. They offer two levels of impact assessment: –first, in terms of how projects provide a business case justification for expenditure and whether they meet the targets set for them, –and second, in terms of how projects meet the goals of the agencies concerned and, in turn, how that helps achieve wider government strategies.

Deepak T. Bhatia26 World Bank and Impact measurement experience What is the WBs experience with measuring impact? Independent evaluation group - IEG measures impact of Bank programs and projects, and draws lessons from experience. The Bank also has a large program of impact evaluation although there may not be any that have centered around the ICT sector. Whether WB prescribes ONE methodology to all countries? How much tailoring is needed? IEG generally uses objective based methodology to measure impact. But if this question refers to tailoring strategies then the key finding from IEG work is that strategies including ICT must be tailored to country circumstances.