TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September 2005, Cyprus

2 the goals of this session To introduce myself and the ZBW To develop a common understanding WHY we are evaluating our work and HOW we are going to do it To present the (preliminary) evaluation concept and to get feedback on it To make sure the words and concepts used are familiar to everybody To develop a common understanding of how to develop criteria that could be used as criteria for success Overview on the upcoming steps

3 Introduction

4 my background Profession: Sociology, further training in the field of evaluation and quality management Researcher in the Center for Education and Economy, Graz, Austria 5 full time employees Public and private clients My main working areas: labour market research, research on vocational education and training, evaluation, statistics and surveys Ca. 20 evaluation projects in the last 4 years

5 project references on EU level Evaluation of the ongoing Leonardo project inclusive training for disability care workers Self-evaluation of the Equal project girls crack IT Evaluation of the Styrian Territorial Employment Pacts Evaluation of the article VI project GO BEST Project management of an ongoing Leonardo project online consulting tool for IT professions Member of a proposal development group in the 6th framework programme, priority 7 Reasearch activities for the CEDEFOP: reports on the Austrian training system, comperative country studies, administrating the ERO database (European Research Overview)

6 workflow of this presentation Evaluation What does it mean? What can it do for us? What do we need? Concept Log Frame Elements of tender Gender Mainstreaming Defining the Log Frame Indicators Terms of Reference Next steps Timeline At this meeting After the meeting Feedback

7 Evaluation >>There are as many different definitions of evaluation as there are evaluators.<<

8 what is evaluation >>Evaluation is the systematic investigation of the merit or worth of an object or process for the purpose of reducing uncertainty in decision making.<< (Mertens, 1998) 3 types of evaluation – regarding the point in time Ex-ante or predictive evaluation concept Acompanying or formative evaluation processes Ex-post or summary evaluation results Different context, different focus Generating and securing knowledge Controlling Legitimizing actions Dialogue with stakeholders, etc…..

9 why evaluation? whats the benefit? An evaluation offers a systematic way of approaching a topic benefits for project planning Gives neutral feedback for all project partners Informs about performance, efficiancy, effectiveness and quality of the project Makes the project results visible marketing, proof Possibility to learn in the runtime of a project Possibility to counteract in case of anticipated problems Secure Knowledge Identify best practice and transferable models

10 what this evaluation should be Support for the transnational cooperation Answering the question: Do we deliver what we are supposed to deliver? Neutral reflexion of what is going on Suggestions for changes and improvement Ensuring positive project development by giving periodic input Fostering the coperation and communication

11 what an / this evaluation should not be A judgement of good and bad, of right or wrong Evaluators are no project outsiders, they are project partners Evaluators are no pitbull terriers, lurking for any mistake Evaluation project controlling No analysis of cost efficiancy No evaluation of the benefits for the target groups

12 What are your expectations?

13 Evaluation concept

14 logical framework One commonly used apporach: logical framework = LogFrame The LogFrame is the starting point of the evaluation Displays the main logic of the evaluation subject A skeleton of the project that has to be evaluated Represents the causal relations between different levels of objectives, actions and results

15 LogFrame GoalsNeeds Ressources / Input Processes / Mechanisms Effects / results Overall objectives Indikators Specific objectives Expected Products Impact Outcome Output Performance Relevance Effectiveness Efficiency Secondary effects - unexpected sustainability Context effects Structure Concept

16 preconditions There are commonly shared, defined and agreed goals Goals have to be measureable There are defined ressources There are defined processes The skeleton of the log frame has to be filled based on information in the kick off stage of the project. Should be kept in mind in every working session of this meeting. Will be worked out by the evaluation team after the meeting in cooperation with the project management

17 call for tenders The call for tender asked for Evaluation of the agreed procedures, tasks and responsibilities formal evaluation Evaluation of the results and the processes to reach these results content evaluation Participation at every meeting Monitoring of the Gender Mainstreaming Process Periodically reporting

18 main elements of the tender Evaluation TCA VENet Formal EvaluationContent Evaluation Concept Evaluation Structural Evaluation Gender Mainstreaming Process Evaluation Result Evaluation VENet Model European Mainsteaming Spport System Internet Platform Final Activity Gender Mainstreaming – cross-sectional topic

19 formal evaluation - concept Concept analysis Assessment of the codified project structure and plan Assessment of management tools Assessment of communication tools Data project documents (e.g. proposal) meetings protocols (e.g. steering group) interviews

20 formal evaluation - structure Structural analysis Focus: transnational cooperation als system of interacting organizations resp. individuals Assessment of flow of communication Assessment of the cooperation patterns Data Hard facts: contribution on formal level (reports, website,..) Soft facts: Social network survey (x2, ) Satisfaction survey (x2, ) Event questionnaire (x5, at every project meeting) Interviews Documents, Internet Platform (?),….

21 Social network analysis (SNA) SNA – research method that focuses on the relation (communication, cooperation, exchange of information,…) between any given set of actors. A SNA survey asks the partners to rate their own network behaviour according to different criterias (frequeny of interaction, channels of interaction, topics of interaction,…). The answers on these questions are producing a picture of the overall network. All individual answers are displayed on the network level. Benefit You get SNA plots of the network structure Key fiugures on well and not well established elements

22 network plots

23 Gender Mainstreaming GM are the efforts we make to ensure that gender aspects and gender equality is considered in all parts and stages of the project (working plan, objectives, actions,…). A cross sectional topic GM aspects are considered in every part of the evaluation and the evaluation itself 3 main GM approaches Equal opportunities approach neutral treatment Diversity approach awareness of the different needs Affirmative action approach focus on women

24 Gender Mainstreaming Formal EvaluationContent Evaluation - Gender sensitive language - Roles and functions - GM as a topic at the meetings - GM as cross sectional topic in the project - Gender aspects in documents - Treatment of GM in general - GM experience of the partners -… - Is GM considered by the partners in - the analysis of underling problems - the definition of goals and target groups - the design of the processes - the implementation - Do the products reflect GM principles? - Design / content - Dissemination - Benefit Data: protocols, interviews, survey questions, products,…

25 Content evaluation Two levels Processes Results Main questions Have the intended results been reached (quantity and quality)? Have the processes to reach these results been carried out as intended? Did the partners contribute as scheduled? Are there any positive or negative deviations? If so, why? Which deductions for the following project work can be made?

26 Content evaluation For each of the five main products VENet model, internet platform,….. Data Documents Interviews The products themselves Check lists resp. progress reports for the project (in cooperation with the project management) Questions in the survey

27 deliverables of the evaluation Short reports before each meeting Distributed electronicly (internet platform) Participation at every transnational meeting (1 w.d.) Presentation of evaluation results and workshop on the findings (1-2,5h) 1 mid-term report 1 final report

28 Feedback on the evaluation model What is your impression? Does the evaluation provide the information we need? GM in our project? How do we take the evaluation results up?

29 Defining the LogFrame

30 Defintion By now the LogFrame is empty Will be based on the transnational project plan with defined goals, milestones and responsibilities Next Step: Filling the LogFrame with information Results from this meeting (e.g. working plan,…) Documents In cooperation with transnational coordination The rough LogFrame will increase in its density as the project goes on

31 Definition The defined aims of the project are crucial for the evaluation! Important: If you set up any aims, goals and objectives: be SMART S…specific: general aims must be broken down M…measurable: no untouchable aims A…acceptance: agreement among all partners R…realistic: no trivial aims, no unrealistic visions T…timely fixed and with fixed responsibilities

32 Definition The LogFrame is the catalogue of the evaluation. Beneath this we will set up the so called Terms of Reference Kind of evaluation handbook, the practice version of the tender Puts down the procedures of the evaluation Commonly shared evaluation principles Will beset up by the ZBW

33 Next Steps

34 At this meeting Consider the LogFrame allready when working out objectives (SMART), products and processes Evaluation grows as the project grows. Some short interviews on the 2nd resp. 3rd day of the meeting – transnational coordinators A concluding event questionaire in the last session of the meeting Define one main contact person regarding the evaluation in each partner organization

35 After the meeting ZBW will work out the LogFrame in cooperation with the transnational coordination Set up the Terms of Reference Set up dates for the next evaluation steps Inform the partners about the evaluation

36 Timeline

37 Your responsibilities! To commit and take active part in the evaluation process To fill in questionnaires carefully To submit documents and information when asked for To return you contribution to the evaluation in time – respect deadlines To evaluate the evaluation! Feedback is welcome!!

38 Contact information Thank you! Lets get it on! Mag. Gerd Beidernikl Zentrum für Bildung und Wirtschaft Joanneumring 5/4, 8010 Graz, Austria (valid 1st October 2005) Mail: Phone: +43 / 316 / Fax: +43 / 316 / Mobile: +32 / 650 /