1. Julian Williams & Maria Pampaka The University of Manchester
‘Mathematical modelling and problem solving’ Evaluation of Instruments for HE-STEM projects
Julian Williams & Maria Pampaka
The University of Manchester

2. Aims to provide evaluation documents and instruments, materials
reflective video on a website for new projects to benefit their own developments.

3. Pilot Instrument development
Attitudes towards modeling
Awareness
Content of this presentation:
How we validated the measures and what it means?
How we can use them to inform practice or progress?

4. The ‘Attitudinal’ Instrument

5. How we validate? [AKA methodology]
Rasch measurement
Rating Scale Model (Likert type items)
Partial Credit Model (mixture of items)
Winsteps software
Evidence for validity:
Fit statistics
Differential Item Functioning
Person-Item Maps

6. Some evidence of validity
Fit statistics […ensuring unidimensionality?]
After deleting these two misfiting items

7. Some evidence of validity
Fit statistics […ensuring unidimensionality?]

8. Some evidence of validity
DIF analysis […ensuring measure invariance …across time]
No statistically significant difference hence measures can be used to compare before and after scores

9. Some evidence of validity
DIF analysis […ensuring measure invariance …across contexts]

10. The Map

11. End result of validation
A measure of disposition to mathematical modeling
A single item for ‘writing a report’

12. Using these measures/item
Matched before and after scores of students in the disposition scale

14. ‘Writing a report’ Item

15. Overall improvement…

16. The ‘Awareness’ Instrument Descriptive statistics of the ‘items’

17. The ‘Awareness’ Instrument Some results from preliminary validation
Fit Statistics…OK

19. Validation… DIF across DPs and courses (significant differences)
OR is it because of marking?

20. Conclusion
The dispositions instruments have some validity for measuring outcomes of disposition towards mm&ps and report-writing (across subsamples/contexts)
The 'awareness' instrument has potential, but needs development: e.g. in how it is used and in the contexts that are used in items.