1. How in the world is that valid & reliable?
Making a case to CAEP
for an alternative way of showing dispositional data quality
Courtney Glazer, Ph.D., Cameron University

2. Objectives
Show EPPs another way to demonstrate quality of some assessment data
Help CAEP learn that multiple methods for demonstrating data quality are widely- accepted among educational researchers
Stop the paradigmatic dissonance!

3. CAEP 3.3 Candidate Quality, Recruitment and Selectivity
EPPs establish and monitor attributes and dispositions beyond academic ability that candidates must demonstrate at admissions and during the program. The provider selects criteria, describes the measures used and evidence of the reliability and validity of those measures, and reports data that show how the academic and non-academic factors predict candidate performance in the program and effective teaching.

4. History of Qualitative Research
1950’s – 1970’s – Advent of postpositivism
Late 1970’s – Qualitative research published in leading journals
1980’s & 1990’s – Multidisciplinary shift expanding from anthropology & sociology
1987 – AERA Qualitative Research SIG founded

5. Postpositivist Paradigm
Ontology – the nature of reality
Relativist = multiple constructed realities
Epistemology – the relationship between the inquirer and the known
Subjectivist = co-created understanding
Methodology – how we gain knowledge of the world

6. Consider Your EPP
What data do you collect that could or should be considered socially constructed?

7. The Blind Men & the Elephant

8. Data Quality
Validity and reliability make sense when there is one objective truth.
These constructs must be redefined to reflect multiple ways of establishing truth (Golafshani 2008, p. 604).
Lincoln & Guba (1985) propose “trustworthiness” as criteria for data quality

9. Establishing Data Quality: Trustworthiness
Credibility – participant’s perspective
Check findings with participants
Review conclusions with peers
Confirmability – corroboration by others
Audit trail that outsiders could follow
Dependability – account for context
Transferability – among contexts
Thick descriptions of context included
(Erlandson, Harris, Skipper & Allen 1993)

10. CAEP Efforts re. Qualitative Methods
Site Visitor Training materials only exclude surveys from the CAEP Sufficient requirements of Data Validity and Data Reliability.
Conversation at CAEPcon Spring 2016
Presentation at CAEPcon Fall 2016 cited Golafshani (2003), but merely used it to define validity
Question asked at CAEP-led state assessment training

11. Dispositional Assessment System
Context
Assessor(s)
Recommendation for Admission
Qualities observed during gen ed courses & initial ed courses
Arts & Sciences faculty
Education faculty
Admission Interview
Qualities expressed in interview setting
Dispositional Assessment
Tutoring individuals/small groups in the field
Practicum Assessment
Working with whole class during 60 hour field exp
School-based faculty
Self-Assessment
Prior to student teaching
Candidate
Student Teaching Assessment
Working with whole class
Exit Survey
At completion of full program

12. Dispositional Assessment System – Data Quality
Credibility – participant’s perspective
Candidate self-assessment twice in program
Assessors discuss results among themselves
Assessors discuss results with candidates
Confirmability – corroboration by others
Dependability – account for context
Data maintained in database
Data reviewed regularly by professional community members
Audit trail reviewed by external auditors
Transferability – among contexts
Context-specific training provided for assessors using specially- designed case studies

13. Consider Your EPP
How is your EPP meeting the criteria of trustworthiness for socially-constructed data?
Credibility – participant’s perspective
Confirmability – corroboration by others
Dependability – account for context
Transferability – among contexts

14. Questions, critiques, & comments are welcome
Courtney Glazer
Director of Educator Preparation
Cameron University
Lawton, Oklahoma