1. 4th Quality Conference (4QC) Impact Assessment Study Rui Sousa, PhD Catholic University of Portugal Lisbon, 16 July 2007

2. Agenda Background to the study Model underlying questionnaire Pilot test results Survey methodology and key dates

3. Background to the Study Goal of Quality Conferences is to improve the quality of Public Services UK Evaluation of Quality Conferences Report –It is difficult to give examples of improvements to service delivery made as a result of attending the QCs. –There should be a greater emphasis on evaluating the QCs and a greater, more structured follow-up of the impact on: Participants work activities as practitioners and decision makers Direct or indirect inspiration for action

4. Questionnaire Development Review of the QCs documents. Review of the literature on Quality Management and Public Services. Exploratory interviews with 4QC participants in Portugal to capture specific context of 4QC. Discussions with DGAEP.

5. Target Respondents Participants in the 4QC: –Public Sector –Non-Profit Organization a not-for-profit organization with public interest goals but which does not belong to the Public Sector –Private Sector an organization which provides services to Public Sector organizations (e.g., information technology, software applications, consulting), but which does not itself provide public services to citizens

6. Underlying Model – Public Sector Participation in 4QC B. Creation of an Environment Conducive to Change C. Increase in the Use of Quality Practices D. Barriers to Action and Suggestions for Improving Effectiveness of Future Conferences A.Respondent Profile

7. B. Environment Conducive to Change Increased knowledge –Ex: Increased knowledge about best practices in public services Motivation for change/improvement –Ex: Inspiration for change Implementation advice –Ex: Obtaining advice for implementing and managing change

8. C. Quality Practices Adapted from EFQM framework: –Top management commitment –Citizen focus –Employee involvement –Process improvement and information management –Formal organization for quality –Social responsibility

9. D. Barriers to action Specific to the organization Specific to the conference

10. Underlying model – Private and Non-Profit Participation in 4QC Improved Support to Public Sector Organizations Suggestions for Improving Effectiveness of Future Conferences Respondent Profile -Increased knowledge of Public Sector needs -Improved relationships -Development of new services

11. Pilot Test Results Respondents Profile 32 respondents (2 July 2007) Public Sector 2990.6% Non-Profit Organization 1 3.1% Private Sector 2 6.3% From 15 different countries Germany - 6 Portugal - 4 Finland, France, Ireland, Latvia, Netherlands, Poland, Slovenia, Spain, Sweden - 2 from each country Greece, Italy, Nigeria, Switzerland - 1 from each country Average completion time 16 min 7 sec

12. Pilot Test Results Respondents Profile Age Under 36: 6 18.8% 36-45: 9 28.1% 46-55: 11 34.4% Over 55 618.8% Female (11 – 34.4%) Male (21 – 65.6%) Intention to participate in the 5QC Confirmed 56.3% Don’t know yet 43.7%

13. Within the public sector Central administration2482.8% Regional administration1 3.5% Local administration413.8% European institutions0 Manager1965.5% Employee1034.5% Pilot Test Results Respondents Profile

14. Selected Results

17. Comments received Positive feedback To improve: –Suggestions for 5th QC organizers Include failed stories – provide potential to learn Soft side (e.g. “curiosity, bewilderment, EU-feeling”) is as much important as the impact measures to get “inspiration” for change –Specific questions What will you expect from the next QC? Suggestions to improve the effectiveness of the conference The questionnaire presumes change is a good thing – this is debatable (…) Too much detail in some questions (e.g.19 items to consider) ?

18. Main Changes Made Removal of the definitions for “Manager” and “Employee”. Question added on “Length of employment with organization”. A few improvements in the wording and sequence of questions. Shortening of section 7 of the questionnaire (2 questions deleted). Open-ended section soliciting suggestions for improving the effectiveness of future QCs.

19. Modified questionnaire: http://asp.infinivista.com/cesop/survey.php?sid=29 (pwd protected) Note: Old pilot test version is still accessible, but will not be acted on.

21. Survey Methodology 1)Obtain and validate list of e-mails of participants a.Via IPSG members b.By sending a letter by post, asking for a valid e-mail address 2) Invitation e-mail sent by web survey software with link to questionnaire. 3) Questionnaire open for 2 weeks - Automated and IPSG-driven follow-up 4) Analysis of results and final report

22. Key Dates 3 August: Validation of e-mail addresses completed. 3 September: Invitation e-mails sent. 3-19 September: Questionnaire active online.

23. Thank you for your collaboration