1. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 5th QualiFLY Meeting Hamburg, 20. 02. 07 Evaluation of a Family Literacy Project in Hamburg Peter May State Institute for teacher training and school developement Hamburg, Germany

2. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Overview 1Internal and external evaluation 2Phases of evaluation 3Applied methods 4First Findings 5Problems

3. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 FörMig Förderung von Kindern und Jugendlichen mit Migrationshintergrund external evaluationinternal evaluation University of Hamburg State Institute, Hamburg Sub-project: HAVASSub-project: Family Literacy diagnosis-based aidparental support Target group: childrenparents and children Form of evaluation: summativeformative Data collection instruments: intelligence testparticipant observation language testinginterviews questionnaire about childrenquestionnaire parents questionnaire educationalist

4. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Phases of evaluation 1.Pre-Investigation of circumstances in the field school year 03/04 Interests (parents, educationalists, headmasters of schools and kindergardens) Circumstances in possible locations 2.Implementation of the project in preschool classes school year 04/05 Analysis of the concept for preschool classes Description of target group Evaluation of the educational concept 3.Consolidation and extension of the concept to first grade classes school year 05/06 Analysis of the concept extension into first grade classes Circumstances in new locations 4.Networking and sustainability School year 06/07 Building up a network with other projects Transitions from preschool to first grade Learning effects of the children

5. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Phases of evaluation 1.Pre-Investigation of circumstances in the field school year 03/04 interests (parents, educationalists, headmasters) circumstances in possible locations Methods: guideline-based interviews with parents from different ethnical and social groups: e.g. families from Turkey, Africa, Russia, German underclass families with educationalists in possible locations (schools, day-care centres) with headmasters from schools and day-care centres

6. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Phases of evaluation 2.Implementation of preschool classes school year 04/05 analysis of the concept for preschool classes description of target group evaluation of the educational concept Methods: participating observation - in introductory events and evening classes for parents - in groups of parents and children in the morning - in advanced training courses for disseminators (teachers, educators) questionnaires, predominantly with open answers - parents at the beginning: motives, circumstances, interests - disseminators: Implementation of the concept at locations questionnaires, predominantly with given answers - parents at the end of the courses: changes (behaviour, attitudes), satisfaction with the courses - teachers: assess the learning development

7. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Phases of evaluation 3.Consolidation and extension of the concept to first grade classes school year 05/06 analysis of the extension of the concept in first grade classes circumstances in new locations Methods: participating observation - in advanced training courses - in courses for parents in first grade classes questionnaires, predominantly with given answers - parents at the beginning: interests, characteristics of the family - parents at the end of the courses: changes, satisfaction - disseminators: - design of the courses - estimation of success testing of the children: - non-verbal intelligence - language competence at the beginning of the course

8. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Phases of evaluation 4.Networking and sustainability school year 06/07 building up a network with other projects transitions from preschool to first grade learning effects of the children Methods: guideline-based interviews with new partners: - other aids in preschools and primary schools - Volkshochschule (language classes for parents) questionnaires: interviews with parents before and after systematic coverage of performances of children: - sequel testing (pre-, post- and follow-up) in preschools, first and second grade (language, reading, writing) - teachers assessment of educational development

9. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Methods of evaluation (1) participating observation - courses for parents - events with parents and children - meetings of disseminators advantages: - direct impressions (atmosphere) - opportunities to get in contact with parents and educationalists - informal conversations with many details disadvantages: - high costly - influence of subjective impressions

10. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Methods of evaluation (2)Oral interview (guideline-based interviews) with headmasters of schools and kindergarten with single disseminators with special experts with single parents (occationally with translators) advantages: - private atmosphere - possibility to adjust the questions spontaneously - offers new, sometimes surprising insights disadvantages: - costly - can be subjectively influenced - difficulty to generalise

11. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Methods of evaluation (3)questionnaires (a) with open answers - disseminators: narration of concepts and procedures collection of suggestions - parents:survey of perceptions collection of wishes required: translations (Turkish, Russian, Farsi, English) advantages: -less costly than interviews - often offers new insights disadvantages: -no systematic coverage of relevant characteristics -partially costly analysis, requires analysis of contents

12. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Methods of evaluation (3)questionnaires (b) with given answers - disseminators: systematic enquiry of circumstances in locations personal backgrounds and motives estimation and assessment -parentsfamilial backgrounds personal characteristics of children changes in behaviour and attitudes (before - after) required: translations advantages: - very economic - systematic coverage of relevant characteristics - statistical analysis possible disadvantages: - reduced answers - sometimes difficult to interpret

13. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Methods of evaluation (4)Testing - intelligence test - language test: language competence at the beginning sequel testing (post, follow-up in first grade) - reading and writing in second grade advantages: - no subjective influence - very economic (group testing) - provides computable data disadvantages: - high input to establish objective circumstances for testing - psychological barriers for parents and educationalists

14. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Outcomes of questioning of parents Number of questionnaires for parents: (8 locations) School year 2004/05: approx. 70; 2005/06: approx. 100; 2006/07: approx. 130 overall: approx. 300 parents Contents of questioning: personal und familial background of parents motives und wishes for participation in courses changes in the families (reading habits, support for children) rate the courses (feed back)

15. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 First language of parents born in Germany: 33 %

16. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 sex 18 % male 82 % female professions professional educated: 59 % employed: 40 % wide broad of professions (from non educated workers to university graduates)

17. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 parents pre-questionnaire: reading and writing at home

18. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 parents pre-questionnaire: reading and writing with the children

19. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Expectations of participants to course attendance 82 % Interest in new ideas and continuation of own education 56 % Support for the own child 42 % Exchanging experiences and to establish contacts 10 % cooperation with teachers

20. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 parents post-questionnaire: reading and writing at home

21. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 parents post-questionnaire: reading and writing with children

22. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 parents post-questionnaire : benefit of the courses for parents

23. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 parents post-questionnaire : benefit of the courses for the children

24. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Evaluation of the courses by the parents 27 % very good 72 % good 1 % satisfying 0 % dissatisfying

25. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07

26. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Disseminators questionnaire:

27. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Disseminators questionnaire:

28. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Disseminators questionnaire:

29. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Disseminators questionnaire:

30. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Summary of evaluation results so far: highly motivated disseminators (teachers and educators) high satisfaction of the parents few changes of parents literacy behaviour more advancement in helping children reading at home benefit for childrens learning in parental view open question : childrens benefit in objective view (tests)

31. Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 Problems with the evaluation serious fluctuations in participation lack of control groups test conditions hard to control analysis of effect conditions complicated, due to complex mass of arrangements.