1. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 1 Consider the Evidence Evidence-driven decision making for secondary schools A resource to assist schools to review their use of data and other evidence 9 Terminology

2. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 2 Terminology Terminology used in the evidence-driven decision making cycle Trigger Clues found in data, hunches Explore Is there really an issue? Question What do you want to know? AssembleGet all useful evidence together Analyse Process data and other evidence Interpret What information do you have? Intervene Design and carry out action Evaluate What was the impact? Reflect What will we change?

3. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 3 Trigger Data, ideas, hunches, etc that set a process in action. The trigger is whatever it is that makes you think there could be an opportunity to improve student achievement. You can routinely scan available data looking for inconsistencies, etc. It can be useful to speculate about possible causes or effects - and then explore data and other evidence to see if there are any grounds for the speculation.

4. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 4 Explore Initial data, ideas or hunches usually need some preliminary exploration to pinpoint the issue and suggest good questions to ask.

5. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 5 Question This is the key point: what question/s do you want answered. Questions can raise an issue and/or propose a possible solution.

6. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 6 Assemble Get together all the data and evidence you might need – some will already exist and some will have to be generated for the occasion.

7. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 7 Analyse Process sets of data and relate them to other evidence. You are looking for trends and results that will answer your questions (but watch out for unexpected results that might suggest a new question).

8. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 8 Interpret Think about the results of the analysis and clarify the knowledge and insights you think you have gained. Interrogate the information. It’s important to look at the information critically. Was the data valid and reliable enough to lead you to firm conclusions? Do the results really mean what they seems to mean? How sure are you about the outcome? What aspects of the information lead to possible action?

9. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 9 Intervene Design and implement a plan of action designed to change the situation you started with. Be sure that your actions are manageable and look at the resourcing needed. Consider how you’ll know what has been achieved.

10. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 10 Evaluate Using measures you decided in advance, assess how successful the intervention has been. Has the situation that triggered the process been improved? What else happened that you maybe didn’t expect?

11. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 11 Reflect Think about what has been learned and discovered – and what practices you will change as a consequence. What did we do that worked? Did this process suggest anything that we need to investigate further? What aspects of the intervention can be maintained? What support will we need?

12. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 12 Terminology Other terms used in Consider the Evidence

13. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 13 Terminology Analysis A detailed examination of data and evidence intended to answer a question or reveal something. This simplistic definition is intended to point out that data analysis is not just about crunching numbers - it’s about looking at data and other evidence in a purposeful way, applying logic, creativity and critical thinking to see if you can find answers to your questions or reveal a need. For example, you can carry out a statistical analysis of national assessment results in the various strands of English across all classes at the same level. You could compare those results with attendance patterns. But you might also think about those results in relation to more subjective evidence - such as how each teacher rates his/her strengths in teaching the various strands.

14. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 14 Terminology Aggregation A number of measures made into one. This is a common and important concept in dealing with data. A single score for a test that contains more than one question is an aggregation - two or more results have been added to get a single result. Aggregation is useful when you have too few data to create a robust measure or you want to gain an overview of a situation. But aggregation can blur distinctions that could be informative. So you will often want to disaggregate some data – to take data apart to see what you can discover from the component parts. For example, a student may do moderately well across a whole subject, but you need to disaggregate the year’s result to see where her weaknesses lie.

15. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 15 Terminology Data Known facts or measurements, probably expressed in some systematic or symbolic way (eg as numbers). Data are codified evidence. (The word is used as a plural noun in this kit.) The concepts of validity and reliability apply to data. It helps to know where particular data came from; how data were collected and maybe processed before you received them. Some data (eg attendance figures) will come from a known source that you have control of and feel you understand and can rely on. Other data (eg standardised test results) come from a source you might not really understand; they may be subject to manipulation and predetermined criteria or processes (like standards or scaling). Some data (eg personality profiles) may be presented as if they are sourced in an objective way but their reliability might be variable.

16. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 16 Terminology Demographics Data relating to characteristics of groups within the school’s population. Data that provides a profile of people at your school. You will have the usual data relating to your students (gender, ethnicity, etc) and your staff (gender, ethnicity, years of experience, etc.). Some schools collect other data, such as the residential distribution of students and parental occupations.

17. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 17 Terminology Disaggregation See aggregation When you disaggregate data, you take aggregated data apart to see what you can discover from the component parts. For example, a student may do moderately well across a whole subject, but you need to disaggregate the year’s result to see where her weaknesses lie.

18. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 18 Terminology Evaluation Any process of reviewing or making a judgement about a process or situation. In this resource, evaluation is used in two different but related ways. After you have analysed data and taken action to change a situation, you will carry out an evaluation to see how successful you have been - this is summative evaluation. But you are also encouraged to evaluate at every step of the way - when you select data, when you decide on questions, when you consider the results of data analysis, when you decide what actions to take on the basis of the data - this is called formative evaluation.

19. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 19 Terminology Evidence Any facts, circumstances or perceptions that can be used as an input for an analysis or decision. For example, the way classes are compiled, how a timetable is structured, how classes are allocated to teachers, student portfolios of work, student opinions. These are not data, because they are not coded as numbers, but they can be factors in shaping teaching and learning and should be taken into account whenever you analyse data and when you decide on action that could improve student achievement.

20. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 20 Terminology Information Knowledge gained from analysing data and making meaning from evidence. Information is knowledge (or understanding) that can inform your decisions. How certain you will be about this knowledge depends on a number of factors: where your data came from, how reliable it was, how rigorous your analysis was. So the information you get from analysing data could be a conclusion, a trend, a possibility.

21. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 21 Terminology Inter-subject analysis A detailed examination of data and evidence gathered from more than one learning area. Inter subject analysis can answer questions or reveal trends about students or teaching practices that are common to more than one learning area. For example, analysing the results of students taking mathematics and physics subjects can indicate the extent to which achievements in physics are aided or impeded by the students’ mathematical skills.

22. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 22 Terminology Intervention Any action that you take to change a situation, generally following an analysis of data and evidence. This term is useful as it emphasises that to change students’ achievement, you will have to change something about the situation that lies behind achievement or non-achievement. You will take action to interrupt the status quo.

23. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 23 Terminology Intra-subject analysis A detailed examination of data and other evidence gathered from within a specific learning area. Intra subject analysis can answer questions or reveal trends about student achievement or teaching within a subject or learning area. For example, an analysis of assessment results for all students studying a particular subject in a school can reveal areas of strength and weakness in student achievement and/or in teaching practices, etc. Comparison of a school’s results in a subject with results in that subject in other schools is also intra subject analysis.

24. www.minedu.govt.nz © New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector. Page 24 Terminology Longitudinal analysis A detailed examination of data and evidence to reveal trends over time. Longitudinal analysis in education is generally used to reveal patterns in student achievement, behaviour, etc over a number of years. Results can reveal the relative impact of different learning environments, for example. In this resource, it is suggested that longitudinal analysis can be applied to teaching practice and school processes. For example, the impact of modified teaching practices in a subject over a number of years can be evaluated by analysing the achievements of successive cohorts of students.