1. © fedict 2005. All rights reserved User satisfaction & impact measurement Belgian practice Description using the Template Christine Mahieu, Business Analyst FEDICT 24 october 2008

2. © fedict 2005. All rights reserved 2 Measurement initiatives 2004 : Fed-eView / Administration Measuring the degree of back-office development of the federal administrations and their egov readiness 2005-2006 : Fed-eView / Citizen Understanding the needs of the citizens 2007 : eGov Measurement Framework Design Based on international best practices Integrated instrument to continuously monitor and benchmark the development of ICT and eGovernment in Belgium Partnerships are essential (Statistics department, regions, municipalities, EU, private partners) 2008 : Implementation of the framework Consolidation of the dispersed existing indicators Development of new indicators and collect data in cooperation with other public and private actors

3. © fedict 2005. All rights reserved 3 Fed-eView Citizen  A strategic decision support tool  To assess the citizen’s needs, perceptions and use of electronic government, covering also ICT usage and security requirements  To monitor trends and evolutions  Methodology : Quantitative survey : Panels from both (internet user /non-user) populations of citizens were followed in several waves over 15 months in 2005- 2006 Qualitative survey : Focus groups (for citizens and for entreprises) provided a list of potential applications to be developed end 2006-begin 2007

4. © fedict 2005. All rights reserved 4 A. General properties  Repeatability  Yes  Independence of size  Maybe, not tested  Estimated cost of using the methodology  About 125.000 euros for 3 waves of survey and 8 focus groups  Prerequisites to carry out and use the measurement  eGov is a horizontal matter, an enabler. Satisfaction about egov services or about public service ?  Stability  Yes, this is the primary objective of the methodology used  Accuracy  Other surveys (like ICT Household survey - urostat) confirm the results  Conformity to WAI and W3C standards  Not a requirement  Number and type of administrations using the method  FEDICT and the Ministry of Economy/DG Statistics for some indicators  Multidimensionality Quantitative survey : 2 types of citizens (internet users and non-users) Qualitative survey : 2 types of users (citizens and entreprises)

5. © fedict 2005. All rights reserved 5 B. Intended use of measurements results 1.Who are the intended users? Policy makers Researchers Public 2.Measurements what for To assess current status and to monitor trends and evolution No measurement on impact

6. © fedict 2005. All rights reserved 6 C. Deployment properties 1.Who carries out the measurement? Outsourced to a neutral third party (team of 3 people + research bureau) 2.How is the measurement carried out? Use of a ´mixed mode survey´with two random samples: An online panel representative for the Belgian Internet population interviewed in 3 waves via CAWI (panel members were invited by e-mail to participate) An offline panel representative for the Belgian internet non-users interviewed in 2 waves via CATI. Frequency is used as the criterion of internet use: a minimum of once a month use of the internet.

7. © fedict 2005. All rights reserved 7 C. Deployment properties 3.Measurement coverage: subset or exhaustive sampling ? The 2 panels constitute representative samples of the Belgian populations of internet users and internet non- users. Note: Belgian population is about 10 millions. Online panel Internet users Telephone panel Internet non-users Wave IN = 3324N = 1421 Wave IIN = 1480N = 624 Wave IIIN = 1139-

8. © fedict 2005. All rights reserved 8 5.Testing  No specific test 8.Score calculation and statistics used  Percentages  Scale of satisfaction : from 0 (absolutely dissatisfied) to 10 (absolutely satisfied)  Use of quadrant for combining indicators : C. Deployment properties

9. © fedict 2005. All rights reserved 9  Comparisons, e.g. time series or needs and wants/actual perception  Yes C. Deployment properties Possession of internet at home 2005 Possession of internet at home 2006 Use of internet 2006 % YesNo 2.2 Yes No2.5 YesNoYes.6 Yes 11.9 No 52.0 NoYesNo1.4 No Yes14.6 NoYes 14.9 Total100.0

10. © fedict 2005. All rights reserved 10 9.Reporting of the evaluation results  The results are summarised in a public paper in 4 languages (on ePractice)  Access to raw data will be possible when the eGov Monitor will be implemented C. Deployment properties

11. © fedict 2005. All rights reserved 11 > Does the measurement lead to policy actions? Partially : definition of citizen personalised one- stop-shop and eBox But no optimum use of the results > Lessons learned Use of panels Low-cost way of citizen’s feedback. Opportunity to go in-depth in the subject Not useful to measure impact of specific communication campaigns Decrease representativity in 15 months time Focus groups: selection of people not easy Measuring the satisfaction of eGov service or of public service ? D. Experience from using the methodology

12. © fedict 2005. All rights reserved 12  Maintained in the context of the global measurement framework eGov Monitor E. Maintenance of Measurement Method