Measuring important learning outcomes in the context of two linked UK (mathematics education) projects: From instrument development, to measure validation and statistical modeling TLRP: “Keeping open the door to mathematically demanding F&HE programmes” (2006 – 2008) TransMaths: “Mathematics learning, identity and educational practice: the transition into Higher Education” ( ) Maria Pampaka (The University of Manchester) Trondheim, February 2011

Outline of Presentation Some background to the projects Instrument Development Measure construction and validation A measure of pedagogy A measure of Mathematics Self Efficacy A modeling approach to respond to the research questions

Funded by ESRC (Economic and Social Research Council) The University of Manchester, School of Education The team: Lead Principal Investigator: Prof Julian Williams Other PIs: –Laura Black –Pauline Davis –Graeme Hutcheson –Brigit Pepin –Geoff Wake Researchers: Paul Hernandez-Martinez Maria Pampaka TLRP: “Keeping open the door to mathematically demanding F&HE programmes” (2006 – 2008) TransMaths: “Mathematics learning, identity and educational practice: the transition into Higher Education” ( )

Educational System in UK (England)

Aims of the Projects TLRP: To understand how cultures of learning and teaching can support learners in ways that help them widen and extend participation in mathematically demanding courses in Further and Higher Education (F&HE) AS Mathematics Vs AS Use of Mathematics TransMaths: To understand how 6th Form and Further Education (pre-university) students can acquire a mathematical disposition and identity that supports their engagement with mathematics in 6fFE and in Higher Education (HE) Focus on Mathematically demanding courses in HE (‘control’ : non mathematically demanding, e.g. Medicine and Education) Mixed methodology: longitudinal case studies, interviews and surveys

March 06 Sept 06 Programme effectiveness Classroom practices Learner identities Questionnaire design Pilot case studies June 07 Sept 07 Dec 07 (i) initial questionnaire (ii) post test (iii) delayed post test Case studies in UoM and traditional AS Follow up case studies (i) initial interviews (ii) interviews round 2 (iii) follow-up interviews DP1 Sept-Nov 2006 DP2 Apr-June 2007 Teachers’ Survey TLRP Research Design: The Survey in the general framework DP3 Sept-Dec 2007

RQ1: How do different mathematics educational practices found in pre- university/university transition interact with social, cultural and historical factors to influence students’ (a) learning outcomes, and (c) decisions in relation to learning and using mathematics? RQ2: How are these practices mediated by different educational systems (their pedagogies, policies, technologies, assessment frameworks, institutional conditions and initiatives)? The Surveys within the TransMaths Project and the Relevant Research Questions

Analytical Framework Instrument Development Measures’ Construction and Validation (Rasch Model) Model Building (Multiple Regression, GLM)

Section A: Background and class information The teacher’s Survey [TLRP]

The TLRP Instrument (Student Questionnaire) Section A: Background Information Section B: [Disposition to enter HE and study mathematically demanding subjects] Section C: Mathematics Self-Efficacy Instrument

Section A: Background Information Name, college, date, date of birth Address and telephone number (for follow up survey/interview) Gender Course (UoM or AS Maths) Previous math qualification (GCSE grade and tier) University attended by close family Language of first choice Education Maintenance Allowance (EMA)

Section B: Dispositions… Disposition to go to HE Intention to go to University? Expectations: family, friends, teachers Disposition to continue with mathematically demanding courses in HE Intention to study more maths after this course? Amount of mathematics in preferred option Importance of amount of mathematics of course in decision Feelings about future study involving maths Preferred type of maths (familiar, new) University: what course? 

Using Mathematics: Self Efficacy A “pure” item: You are asked to rate how confident you are that you will be able to solve each problem, without actually doing the problem, using a scale from 1(=not confident at all) to 4(= very confident)

Using Mathematics: Self Efficacy An “applied” item:

The TransMaths Student Questionnaire Section A: Background Information –University, Course/Programme –Previous math qualifications –Ethnicity, gender, country of origin, language –Proxies of socio economic background –Special educational needs A series of instruments about different aspects of the transition to HE…

Items / MeasuresDP4DP5DP6 Reasons for choosing University and course Experiences that influence choice of Uni Programme Disposition to complete chosen course Preparedness and Usefulness of ways of studying Transitional Experiences Mathematics Dispositions Perceived Pedagogic Practices at Pre-uni (maths) experience Perceived Pedagogic Practices at Uni (maths) Mathematics Self Efficacy Confidence with Mathematics Usefulness of Mathematics Perceived Mathematical support at Uni Relevance of mathematics The TransMaths ‘instruments’

Analytical Framework Instrument Development Measures’ Construction and Validation (Rasch Model) Model Building (Multiple Regression, GLM)

Constructing the measures: Measurement methodology ‘Theoretically’: Rasch Analysis – Partial Credit Model – Rating Scale Model ‘In practice’ – the tools: –FACETS and Quest Software [Winsteps more user friendly] Interpreting Results: –Fit Statistics (to ensure unidimensional measures) –Differential Item Functioning for ‘subject’ groups –Person-Item maps for hierarchy

Example 1: Measuring Mathematics Self Efficacy…some background Self-efficacy (SE) beliefs “involve peoples’ capabilities to organise and execute courses of action required to produce given attainments” and perceived self-efficacy “is a judgment of one’s ability to organise and execute given types of performances…” (Bandura 1997, p. 3) "a situational or problem-specific assessment of an individual's confidence in her or his ability to successfully perform or accomplish a particular maths task or problem" (Hackett & Betz, 1989, p. 262)

Background – Why Mathematics Self Efficacy? ‘Important in students’ decision making (sometimes more than actual test scores) Positive influence on students’ academic choices, effort and persistence, and choices in careers related to maths and science. How to measure? Contextualised questions TLRP project: a 30 item instrument for pre-university students

Mean plots for the three MSE measures by course and Data Point Result from TLRP (Overall measure, and 2 subscales)

 Instrument measuring students’ confidence in different mathematical areas (  10 items): Calculating/estimating Using ration and proportion Manipulating algebraic expressions Proofs/proving Problem solving Modelling real situations Using basic calculus (differentiation/integration) Using complex calculus (differential equations / multiple integrals) Using statistics Using complex numbers Example 1: Measuring Mathematics Self Efficacy [at the transition to university]

An example Item … Measuring Mathematics Self Efficacy An example “applied” item

Methods and Sample 10 items 4 point Likert Scale (for frequency) Sample:1630 students Rasch Rating Scale Model

Item Fit Statistics t check for the assumption of unidimensionality Results [1] – Checking Validity One measure?

Items more relevant to AS/A2 Maths context More difficult for non maths students Differential Item Functioning Results [2] – Checking validity Differences among student groups

Multidimensional Scaling? Results [3] Two separate measures

Constructing the measures Example 2: The Teacher Survey (TLRP) ‘ 28 item survey to teachers  5 point Likert Scale (for frequency)  Sample:110 cases from current project  Rasch Rating Scale Model

B6: I encourage students to work more slowly B24: I cover only the important ideas in a topic Constructing the measures – Validity [Unidimensionality - Fit]

30 Students work on their own, consulting a neighbour from time to time. Students [don’t] compare different methods for doing questions. Students [don’t] work on substantial tasks that can be worked on at different levels. I try to cover everything in a topic. Students work through exercises. Constructing the measures – Validity across different groups (DIF)

Constructing the measures: A measure of ‘pedagogical style’

I tend to follow the textbook closely Students (don’t) discuss their ideas I encourage students to work more quickly I know exactly what maths the lesson will contain Students (don’t) invent their own methods I tell students which questions to tackle I teach each topic separately Constructing the measures: A measure of ‘pedagogical style’

“It’s old fashion methods, there’s a bit of input from me at the front and then I try to get them working, practicing questions as quickly as possible…” “… there’s a sense that I’ve achieved the purpose…I’ve found out what they’ve come with and what they haven’t come with so…we can work with that now” “…. from the teachers that I’ve met and talked to… it seems to me that one of the big differences is, I mean I don’t sort of use textbooks… [ ]…I want to get students to think about the math, I want students to understand, I want students to connect ideas together, to see all those things that go together and I don’t think a text book did that…[ ]. Validation supported by qualitative data “…I do tend to teach to the syllabus now…If it’s not on I don’t teach it. … but I do tend to say this is going to be on the exam…”

Transitional Experiences The most challenging measure…

Analytical Framework Instrument Development Measures’ Construction and Validation (Rasch Model) Model Building (Multiple Regression, GLM)

From measures to GLM Modeling - TLRP Variables Outcome of AS Maths (Grade, or Dropout) Background Variables Disposition Measures at each DP –Disposition to go into HE (HEdisp) –Disposition to study mathematically demanding subjects in HE (MHEdisp) –Maths Self Efficacy (MSE: overall, pure, applied) A score of ‘pedagogy’ based on teacher’s survey

Longitudinal design –DP1: 1792 –DP2: 1082 –DP3: 608 Resolution for some outcome variables (e.g. AS outcome) –Phone survey, School’s databases, Other databases The TLRP Sample

Results [1]: Math Dropouts Percentages of dropouts by course and previous attainment Effect Plots for a logistic regression model of dropout

Positive effect: Math Disposition at DP1, Maths Self- Efficacy and ‘Mathematical demand of other subjects’ Results [2]: A Model for HE Maths Disposition at DP2

Negative effect of Pedagogy

Our modeling framework (TransMaths)

We hypothesized that: Outcome of Year 1 (at University)= Entry Qualification + Dispositions + Transitional experiences + Background Variables An example Model (TransMaths)

The resulting Linear Regression Model Positive effect: Positive transitional experience, previous maths qualification Negative effect: Course, Low Participation Neighbourhood, Mathematics Self Efficacy

Example GLM A model of mathematics disposition at end of first year university based on variables: Gender International (Yes / No) Special Educational Needs (SEN) “Mathematical” = Mathematically demanding courses (Maths + Engineering) Subject Area

Notes: SEN: Mainly Dyslexia Subject Area reference category: Engineering Modeling Maths Disposition at DP5

Similar Model from our previous TLRP project DP1: Beginning of AS DP2: End of AS

This is the quantitative aspect of mixed method project which also includes case studies and interviews The multi-step methodology described helps us to create and validate our measures and then… Use them to model (GLM- regression modeling) in order to respond to the research questions we originally set… For instance What influences students successful transitions between various stages of education (e.g. to University)? How can we predict students’ progress at university) ? Just a short summary

Thank you Q & A