Julian Williams & Maria Pampaka The University of Manchester

Slides:

Advertisements

Similar presentations

How valuable is Source A as evidence of How useful is Source A as evidence of Assess the value of Source A as evidence of How useful is Xs account (Source.

Advertisements

Evaluation of a process for reflection on feedback to support student learning Mark M c Crory Steve M c Peake Denise Currie Department of Management and.

How to write a L3 EVALUATION

Evaluation at The Prince’s Trust Fire Service Prince's Trust Association meeting 18 th February 2010 Subtitle.

Designs to Estimate Impacts of MSP Projects with Confidence. Ellen Bobronnikov March 29, 2010.

Experimental Research Designs

Do Patients Assess Themselves Differently to Clinicians? Validation of the Lysholm Knee Scale Heather Smith Clinical Trials Manager RJAH Orthopaedic Hospital,

A controversy in PISA and other large- scale assessments: the trade-off between model fit, invariance and validity David Andrich CEM: 30 years of Evidence.

Overview of field trial analysis procedures National Research Coordinators Meeting Windsor, June 2008.

Measuring Pedagogies from Secondary School to University and Implications for Mathematics education (in UK and abroad) Maria Pampaka & Julian Williams.

© UCLES 2013 Assessing the Fit of IRT Models in Language Testing Muhammad Naveed Khalid Ardeshir Geranpayeh.

1 Reducing the duration and cost of assessment with the GAIN: Computer Adaptive Testing.

Measurement Problems within Assessment: Can Rasch Analysis help us? Mike Horton Bipin Bhakta Alan Tennant.

Analysis of Preschool Assessment Data Desired Results Development Profile Preschool © DRDP – PS (2010) Ifthika “Shine” Nissar, M.A.

Identification of Misfit Item Using IRT Models Dr Muhammad Naveed Khalid.

NCCSAD Advisory Board1 Research Objective Two Alignment Methodologies Diane M. Browder, PhD Claudia Flowers, PhD University of North Carolina at Charlotte.

The Project Planning Process

Moving from Development to Efficacy & Intervention Fidelity Topics National Center for Special Education Research Grantee Meeting: June 28, 2010.

Name of School Name of Portfolio Grade Level Inclusive Dates.

Teaching and Learning Practices in Secondary Mathematics: measuring teaching from teachers’ and students’ perspective Maria Pampaka, Lawrence Wo, Afroditi.

Measuring Mathematical Knowledge for Teaching: Measurement and Modeling Issues in Constructing and Using Teacher Assessments DeAnn Huinker, Daniel A. Sass,

Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham.

Crossing Methodological Borders to Develop and Implement an Approach for Determining the Value of Energy Efficiency R&D Programs Presented at the American.

1 Milwaukee Mathematics Partnership Program Evaluation Year 6 Results Carl Hanssen Hanssen Consulting, LLC Cindy Walker University of Wisconsin-Milwaukee.

Biology CPD Presentation Aims To supplement previously published information about the internal assessment of Biology Units at National 3, 4 and 5 To.

March 26-28, 2013 SINGAPORE CDIO Asian Regional Meeting and Workshop on Engineering Education and Policies for Regional Leaders Programme Evaluation (CDIO.

Building on Transmaths: TeLePriSM Teaching and Learning Practices in Secondary Mathematics Maria Pampaka 2 nd March 2012 Formal ESRC Title: “Mathematics.

What is the effect of self- efficacy on students’ participation? Investigators: Julian Williams (PI), Laura Black, Pauline Davis, Birgit Pepin and Geoff.

Career Field Research Project 3 points => Journal Articles Word document submitted on Moodle 1 Point => Independent Reflection Word document submitted.

University Teaching Symposium January 9,  Individual Development and Educational Assessment  Kansas State University  A diagnostic to.

SURVEY RESEARCH.  Purposes and general principles Survey research as a general approach for collecting descriptive data Surveys as data collection methods.

New Advanced Higher Subject Implementation Events Statistics Unit Assessment at Advanced Higher.

Community Service Portfolio Name of School and Class Name of School Grade Level Inclusive Dates.

Assessment seminar 2010 Accounting and Finance 2010/05037.

A course is designed to increase mathematical comprehension. In order to evaluate the effectiveness of the course, students are given a test before and.

What Are the Characteristics of an Effective Portfolio? By Jay Barrett.

Evaluating Impacts of MSP Grants Ellen Bobronnikov January 6, 2009 Common Issues and Potential Solutions.

Faculty Information Form Insight Improvement Impact ® University of Alabama Birmingham.

MAP: Measured Academic Progress© Parent Coffee February 10, 2010.

Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov February 16, 2011.

Measuring important learning outcomes in the context of two linked UK (mathematics education) projects: From instrument development, to measure validation.

Aligning Assessments to Monitor Growth in Math Achievement: A Validity Study Jack B. Monpas-Huber, Ph.D. Director of Assessment & Student Information Washington.

 Youth Teasing and Bullying are a major public health problem  ~20% of youths report being bullied or bullying at school in a given year  160,000.

Prior Learning Assessment (PLA) Pilot Project At VSU Prepared by the PLA Assessors Group.

Measuring Mathematics Self Efficacy of students at the beginning of their Higher Education Studies With the TransMaths group BCME Manchester Maria.

Improving Data Collection through Rasch Measurement: A Continuing Study of Teacher Supply and Demand Kelly D. Bradley and Shannon O. Sampson University.

Copyright © 2014 American Institutes for Research and Cleveland Metropolitan School District. All rights reserved. March 2014 Interpreting Vendor Assessment.

Quantification of dyspnea using descriptors: Development and initial testing of the Dyspnea-12 J Yorke, S H Moosavi, C Shuldham, P W Jones (Thorax

OFFICE OF EDUCATION Consultations – 13 August 2014 Work to date related to the ELP loading.

Copyright © Springer Publishing Company, LLC. All Rights Reserved. EVIDENCE-BASED TEACHING IN NURSING – Chapter 15 –

EVALUATING EPP-CREATED ASSESSMENTS

Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov March 23, 2011.

Cari-Ana, Alexis, Sean, Matt

PROGRAM & POLICY EVALUATION ‘Goal Attainment Scaling’

Assessment Research Centre Online Testing System (ARCOTS)

Pilot Test and Field Trial at TUS Results

Evidence for gender bias in interpreting online professor ratings

(with the Teleprism team)

PRMSP/AlACiMa Project

Scaling up inclusive education within STEM: Sharing experiences across HE Samantha Child Kate Lister

Teaching and Learning Practices in Secondary Mathematics

STTEP project Supporting Transition Towards Education Progression

X-STANDARD MATHEMATICS ONE MARK QUESTIONS

Mathematics/Science/Health Methods for ECE/Elem/MS Teachers

Student Satisfaction Results

Project collaborators’ names

Conclusions/ Future Directions

Science Network Pilot Evaluation for …School

An Empirical Study of Learning Strategy Use by Differently Proficient Students in a Web-based Environment Wang Zhiru.

An overview of course assessment

Presentation transcript:

Julian Williams & Maria Pampaka The University of Manchester ‘Mathematical modelling and problem solving’ Evaluation of Instruments for HE-STEM projects Julian Williams & Maria Pampaka The University of Manchester

Aims to provide evaluation documents and instruments, materials reflective video on a website for new projects to benefit their own developments.

Pilot Instrument development Attitudes towards modeling Awareness Content of this presentation: How we validated the measures and what it means? How we can use them to inform practice or progress?

The ‘Attitudinal’ Instrument

How we validate? [AKA methodology] Rasch measurement Rating Scale Model (Likert type items) Partial Credit Model (mixture of items) Winsteps software Evidence for validity: Fit statistics Differential Item Functioning Person-Item Maps

Some evidence of validity Fit statistics […ensuring unidimensionality?] After deleting these two misfiting items

Some evidence of validity Fit statistics […ensuring unidimensionality?]

Some evidence of validity DIF analysis […ensuring measure invariance …across time] No statistically significant difference hence measures can be used to compare before and after scores

Some evidence of validity DIF analysis […ensuring measure invariance …across contexts]

The Map

End result of validation A measure of disposition to mathematical modeling A single item for ‘writing a report’

Using these measures/item Matched before and after scores of students in the disposition scale

‘Writing a report’ Item

Overall improvement…

The ‘Awareness’ Instrument Descriptive statistics of the ‘items’

The ‘Awareness’ Instrument Some results from preliminary validation Fit Statistics…OK

Validation… DIF across DPs and courses (significant differences) OR is it because of marking?

Conclusion The dispositions instruments have some validity for measuring outcomes of disposition towards mm&ps and report-writing (across subsamples/contexts) The 'awareness' instrument has potential, but needs development: e.g. in how it is used and in the contexts that are used in items.