WP4: Outcomes from Trying out Exemplars Avi Hofstein, Rachel Mamlok-Naaman, Ron blonder & Mira Kipnis The Weizmann Institute of Science Rehovot 76100, Israel PARSEL Meeting, Lund, august 2007

Main Goal “WP4 follows on from WP3 and relates to outcomes from the use of the materials and/or resources nationally.” “Main goal: Determination of outcomes from trying them and finding out whether the materials and/or resources are usable as intended, and can function in the desired approach based on the model. The determination of outcomes will be undertaken by a variety of methods and guided by the national network partner….

Main Goal The national network partner ensures wider dissemination within the country and to the project dissemination team through the website, local seminars, journal articles and videoconferencing. This work package will be managed by the Weizmann Institute.” (PARSEL document, 2006)

What are we aiming at? Modules (popular, relevant, student / teacher centred, context-based…) Teachers (adaptation and assimilation) Students (change of attitudes and beliefs towards science…)

The PARSEL Evaluation Instruments Introduction and Rationale This evaluation draft relates to written responses and to optional interviews for teachers and for students, and its objectives are to assess the following: Was the module perceived by the students as relevant to their lives? Was the module interesting and useful? Did the module enhance the students' scientific literacy? Was the module student centered? Did the module develop cognitive skills? How was the module used (formative aspects)?

It has already been proposed and accepted that the evaluation will come from the teacher and the students involved in the trying out of modules. Thus two sets of instruments are needed. But at the same time, it is important to recognize that the instruments should not be extensive, taking away valuable teaching time or involving the teachers and/or students in extensive out- of-school tasks.

A further consideration is that the instruments are for the purpose of finding out about the usefulness of the modules at a pan European level. The evaluation is not a research study, although any partner can build on the work of the consortium and, for their purposes, go beyond the project and undertake additional evaluation.

It is proposed that it is reasonable for each teacher involved in the trying out (hereafter called a PARSEL teacher) to be asked to complete 2 instruments only. Furthermore, it is proposed that each student involved in the trying out - that is taught by a PARSEL teacher – also be involved in completing 2 instruments.

4 instruments For the PARSEL teacher: A teacher ownership questionnaire (we have another alternative – suggested by Wolfgang) An interview at the end of the trying out For the students: A pre-questionnaire describing prior teaching / learning science (we have another alternative – suggested by Claus) A post – questionnaire after each module undertaken (we have other 2 alternatives) An optional interview at the end of the trying out (to triangulate the quantitative aspects)

PARSEL Evaluation Instruments: Three sections SECTION 1 Feedback of how the teacher used the module (to be completed by the teacher during or immediately after teaching of the module) SECTION 2 Pre and Post Semantic differential (SD) questionnaires for students SECTION 3 Examples of interview questions (items) for teachers and students

SECTION 1. Feedback on how the teachers used the module The objectives of this section is to obtain feedback from the teacher on how they took ownership of the PARSEL module and in so doing how they modified the various components of the module (if at all), and how they covered each stage. The teachers are asked to complete components as the teaching proceeds and thus the work load is distributed over the whole teaching time for the module.

SECTION 2A. Student Pre-questionnaire: Students' opinions related to the studying of science This draft relates to student responses to a semantic differential questionnaire. Based on literature, a semantic differential questionnaire is a simple and precise instrument to investigate attitudes, beliefs and feelings toward certain issues. Osgood, C.E., Suci, G.J., & Teenenbaum, P.H. (1957). The measurement of meaning. Urbana, IL: University of Illinois Press

Main goal of the semantic differential questionnaire How to strive for scientific literacy whilst making the teaching of science both popular and relevant to students? Interest/enjoyment/fun for students Important/meaningful/useful as perceived by students Cognitive /NOS /personal /social challenges in student learning, relevant from a society/career/citizen perspective (currently and in the future) Student involvement/varied/simulating approaches (e.g. scenario), enquiry/experimental/student centered / providing feedback to the teaching approach.

Objectives of the semantic differential questionnaire Was the module relevant to students' lives? (1, 13, 14, 17) Was the module interesting? (3, 5, 16, 18) Did the module enhance the students' scientific literacy? (10, 4, 12, 19) Was the module student centered? (7, 9, 15, 20) Did the module develop cognitive skills? (2, 6, 8, 11)

Objectives of the semantic differential questionnaire To assess: For items 1-20: Did students perceive the teaching in science lessons as being relevant to their lives? Was the teaching of science interesting and useful? Did the prior teaching enhance the students’ scientific literacy? Was the teaching student centered? Was the teaching perceived as developing cognitive and affective (interest, motivation) skills?

For items 21-24: Did the teaching / learning of science provide opportunities for group work? Did the teaching encourage inquiry learning? Include assessment for learning (formative assessment)? Include provision for giving a meaning to science (appreciating the nature of science - NOS)?

SECTION 2B. Student Post-questionnaire, to be administered to the students after the completion of each module - Objectives To assess the following: For items 1-20 Did students perceive the teaching in science lessons as being relevant to their lives? Was the teaching of the module interesting and useful? Did the teaching enhance the students’ scientific literacy? Was the teaching student centered? Was the teaching perceived as developing cognitive skills?

For items 21-24: Did the teaching provide opportunities for group work? Did the teaching encourage inquiry learning? Did the teaching include assessment for learning (formative assessment)? Did the teaching include provision for giving a meaning to science (appreciating the nature of science - NOS)?

SECTION 3A. Semi-structured interview questions for teachers who are interviewed by the PARSEL personnel This is taken as being interviews with the teachers after teaching at least one module and preferably after teaching a number of modules. This set of interview questions are really only indirectly related to popularity and relevance but do put stress on seeking teacher reactions to the use of the modules. It will be interesting to see how they react to modules which have not been designed in their country and hence relate to another education system.

Purpose of the questions Question 1: Why did you choose to become a PARSEL teacher? To determine whether the teacher has prior appreciation of the intention of PARSEL and/or the style of the modules.

Question 2: What were your feelings before, during and after teaching this module? A major attempt to get at the teacher reactions to the modules. Unfortunately external pressures (modules not related to the examination questions) can be expected to interfere with a ‘real’ reaction, but nevertheless it is expected to give a good indicator of the ‘worth’ of the module or modules in the eyes of the teacher

Question 3: What was your impression regarding the students' different reactions to this module? Allows the teacher to comment on their perception of students’ appreciation of the modules. The inclusion of the term ‘different’ is to push the teacher to recognize that a simple global response is not intended and that a more in-depth response to different student reactions is needed.

Question 4: Will you use this kind of material in the future? Intends to get a clear response on whether the teacher likes the modules and feels they are worthy of inclusion in the teaching schedule. Teacher responses can still be clouded by the actual teaching situation in the school and pressures from external sources e.g. examinations and hence the black and white response may not be forthcoming.

Question 5: What importance do you give to the wider goals of science education prormoted through these modules? To determine the ‘readiness’ of the teacher for science teaching that goes beyond the cognitive and gives an indication of how far the teacher is willing to embrace wider education goals. For some teachers, used to following the textbook as the major or sole source of guidance, this question may have no meaning and hence, for them, unanswerable.

Question 6: Do you feel this project promoting similar materials across the whole of Europe and Israel is worthwhile? To find out the teacher’s interest and reaction to science education outside the country and hence whether they have an interest in knowing more about science education outside the country.

Question 7: Will you recommend the module to your colleagues? To validate all the above.

SECTION 3B. Semi-structured interview questions for students who are interviewed by the PARSEL personnel This section is optional. We believe that a lot of data can be gained through interviews. We found that such data may validate the quantitative results. However, if it puts too much pressure on the teachers and on the PARSEL personnel, it may be skipped.