Workshop, 29 of june Evaluation tools and criteria for eGov services _________________ Leda Guidi Responsible of Communication services to citiezens - Iperbole civic network BOLOGNA MUNICIPALITY Knowledge Society Forum - Telecities Summer Event Bologna, June 2007 Comune di Bologna

Knowledge Society Forum - Telecities Summer EventBologna, June workshop aim The aim of the workshop is to compare different evaluation systems of e-gov services sustainability in order to share approaches and methodologies in: ROI (qualitative and quantitative) measurement definition of indicators (financial, social, political, etc.) achievements of success as well as weak points Bologna will present the tool developed in partnership with Deloitte within the framework of the EUROCITIES «e-Citizenship for All» prize.

Knowledge Society Forum - Telecities Summer EventBologna, June eGov in Bologna The Bologna Municipality has always been open to the use of the most advanced information and communication means both in the administrative and internal managing action as well as in the relation with citizens and the local community. Through the last ten years the Municipality has been constantly focusing its government action towards a considerable increase in the use of new networking technologies for the delivery of information and services to community, the two way communications and participation trials. Despite the efforts spent to fight digital barriers, a part of Bologna population still risks to be cut off from e-participation processes (age, gender, social-economic situation, etc.)

Knowledge Society Forum - Telecities Summer EventBologna, June To make citizens proactive and fully aware of its e-rights the key is to invest in e-Citizenship at all levels of multichannel delivery: services information, interaction, participation in decision making processes.  Local public administrations need to build consensus around shared choices, based on wider community/ies participation in the ongoing decision-making processes. A success factor is inter-institutional, multi-level cooperation, partnerships, in order to achieve resource effectiveness generate synergies standardise approaches, formats and languages eGov and eCitizenship

Knowledge Society Forum - Telecities Summer EventBologna, June What “business model” for e-gov? Sustainability for e-service and e-democracy Impact at a medium/long term New methodologies to “measure” the e-gov benefits The efficacy and efficiency of an eGov project is not measurable in the short term (break even!) Only after months / years it becomes possible to give estimations of its impact on the social growth of the community and evolution. It is necessary to use techniques more related to “social balance” than to “analytic book-keeping”. Complex mix of indicators both qualitative and quantitative to evaluate the ROI also in term of innovation eGov Evaluation

Knowledge Society Forum - Telecities Summer EventBologna, June Aim of the Assessment System The aim of the tool developed is to provide the Municipality with a qualitative indicator that will support assessment of actions required to implement e-government. The final indicator value is obtained through assessment of the following drivers :  Organizational  Economic  Social  Financial  Qualitative (vis-à-vis the user)

Knowledge Society Forum - Telecities Summer EventBologna, June Content Index Introduction Aim of the Assessment System Index of Content Method Adopted Guidelines Qualitative Assessment Criteria Qualitative Indicators Assessment Drivers Comments Economic Assessment Criteria Economic Indicators Overall Assessment of the Initiative Final Score Assessment Tool IT-based Assessment

Knowledge Society Forum - Telecities Summer EventBologna, June Method adopted The Assessment Model to evaluate e-government initiatives entails the following steps: Defining a set of e-gov project objectives: assessment indicators Defining a scale of values so as to assign a score system to each driver Identifying assessment drivers for each indicator Developing a computerised system for the automatic calculation of a summary assessment index. Developing a questionnaire regarding the qualitative aspects of the initiative

Knowledge Society Forum - Telecities Summer EventBologna, June Guidelines The assessment criteria must be deployable for any e-government initiative; The system must be able to take into account the Administration’s weighting of the various achievable end points; The system must be able to produce a result that is weighted for the importance attributed by the Administration to the financial impact of the e-gov initiatives; The indicators must be assessed by attributing value judgements to the various assessment drivers; All value judgements must be motivated; The indicator must be measurable by means of a scoring system; The system should be operational without excessive staff requirements; The final overall assessment should be concise and accurate. The guidelines underpinning the assessment system are as follows:

Knowledge Society Forum - Telecities Summer EventBologna, June Qualitative Assessment Criteria

Knowledge Society Forum - Telecities Summer EventBologna, June Qualitative Indicators Qualitative Assessment Criteria Qualitative indicators derive from the objectives the Administration intends to achieve with e-gov initiatives: Image Initiative assessment Economic savings Advantages for society Quality of services Performance

Knowledge Society Forum - Telecities Summer EventBologna, June Assessment Drivers Qualitative Assessment Criteria Qualitative indicators are evaluated by assessment drivers: Type of technology used Cutting throughput times Cutting waiting (queues) Potential to replace “traditional” procedure Quality of services Cutting user transfers Increased transparency/user oversight of administrative practices Integration of disadvantaged groups Economic development of disadvantaged areas Employment gains Advantages for society Potential to replace workforce Potential to reduce workstations Potential to reduce use of materials Potential to reduce public counters Economic savings Potential to cut resource requirements Potential to reduce throughput times Potential to reduce workload Potential to reduce errors Service performance User awareness of subject matter Press awareness of subject matter User pool for the service offered Degree of “sophistication” of service offered Attention capturing capacity Administration’s image ASSESSMENT DRIVERINDICATOR

Knowledge Society Forum - Telecities Summer EventBologna, June Score Systems and Weightings Qualitative Assessment Criteria Drivers are qualitatively assessed, each judgement corresponding to a value Depending on driver evaluation, indicators will have a score ranging from 0 to 10 and will also be weighted for the importance attributed by the Administration Weighting attributed by Administration to e-gov objectives PWeightin g Enhance Administration’s image 0,2 Enhance service performance 0,2 Obtain economic saving 0,2 Increase quality of service 0,2 Advantages for Society 0, Enhance Administration’s image Enhance service performance Obtain economic saving Increase quality of service Advantages for Society With the weighting system, overall assessment of qualitative aspects is expressed by a score from 0 to 10 The sum of the weighting must be 1

Knowledge Society Forum - Telecities Summer EventBologna, June Economic Assessment Criteria

Knowledge Society Forum - Telecities Summer EventBologna, June Economic Indicators Economic Assessment Criteria Economic indicators consider the financial requirement for initiative implementation:  Computer systems  Office equipment and furniture  General plant and equipment  Operational and management  costs  Training Cost for: Total financial requirement assessed by scoring system From:ToSCORE oltre1 Economic aspects expressed by a scoring system from 0 to 10

Knowledge Society Forum - Telecities Summer EventBologna, June Overall Assessment of the Initiative

Knowledge Society Forum - Telecities Summer EventBologna, June Final Score Overall Assessment of the Initiative The final score is the sum of the overall qualitative assessment and the economic assessment, weighted for the importance attributed by the Administration to the economic factors Economic Score Economic Value X Weighting attributed (from 0 to 1) + Qualitative Score Qualitative Value X Weighting attributed (from 0 to 1) The sum must be 1 = Overall assessment of the initiative Score range: 1 to 10

Knowledge Society Forum - Telecities Summer EventBologna, June Assessment Tool

Knowledge Society Forum - Telecities Summer EventBologna, June IT-based Assessment Assessment Tool A computer spread sheet automatically attributing a value to each indicator Economic Aspect Qualitative Aspect  Input of estimated financial investment requirement  Input of Administration’s weighting of financial aspect compared to qualitative aspect  Value judgement given to each driver indicated by putting “X” in questionnaire fields provided.  Value judgement is motivated  Inclusion of weighting for each indicator Each initiative receives an overall score (range 0 to 10) that will support the Municipality in assessing action to be implemented