How in the world is that valid & reliable? Making a case to CAEP for an alternative way of showing dispositional data quality Courtney Glazer, Ph.D., Cameron University

Objectives Show EPPs another way to demonstrate quality of some assessment data Help CAEP learn that multiple methods for demonstrating data quality are widely- accepted among educational researchers Stop the paradigmatic dissonance!

CAEP 3.3 Candidate Quality, Recruitment and Selectivity EPPs establish and monitor attributes and dispositions beyond academic ability that candidates must demonstrate at admissions and during the program. The provider selects criteria, describes the measures used and evidence of the reliability and validity of those measures, and reports data that show how the academic and non-academic factors predict candidate performance in the program and effective teaching.

History of Qualitative Research 1950’s – 1970’s – Advent of postpositivism Late 1970’s – Qualitative research published in leading journals 1980’s & 1990’s – Multidisciplinary shift expanding from anthropology & sociology 1987 – AERA Qualitative Research SIG founded

Postpositivist Paradigm Ontology – the nature of reality Relativist = multiple constructed realities Epistemology – the relationship between the inquirer and the known Subjectivist = co-created understanding Methodology – how we gain knowledge of the world

Consider Your EPP What data do you collect that could or should be considered socially constructed?

The Blind Men & the Elephant

Data Quality Validity and reliability make sense when there is one objective truth. These constructs must be redefined to reflect multiple ways of establishing truth (Golafshani 2008, p. 604). Lincoln & Guba (1985) propose “trustworthiness” as criteria for data quality

Establishing Data Quality: Trustworthiness Credibility – participant’s perspective Check findings with participants Review conclusions with peers Confirmability – corroboration by others Audit trail that outsiders could follow Dependability – account for context Transferability – among contexts Thick descriptions of context included (Erlandson, Harris, Skipper & Allen 1993)

CAEP Efforts re. Qualitative Methods Site Visitor Training materials only exclude surveys from the CAEP Sufficient requirements of Data Validity and Data Reliability. Conversation at CAEPcon Spring 2016 Presentation at CAEPcon Fall 2016 cited Golafshani (2003), but merely used it to define validity Question asked at CAEP-led state assessment training

Dispositional Assessment System Context Assessor(s) Recommendation for Admission Qualities observed during gen ed courses & initial ed courses Arts & Sciences faculty Education faculty Admission Interview Qualities expressed in interview setting Dispositional Assessment Tutoring individuals/small groups in the field Practicum Assessment Working with whole class during 60 hour field exp School-based faculty Self-Assessment Prior to student teaching Candidate Student Teaching Assessment Working with whole class Exit Survey At completion of full program

Dispositional Assessment System – Data Quality Credibility – participant’s perspective Candidate self-assessment twice in program Assessors discuss results among themselves Assessors discuss results with candidates Confirmability – corroboration by others Dependability – account for context Data maintained in database Data reviewed regularly by professional community members Audit trail reviewed by external auditors Transferability – among contexts Context-specific training provided for assessors using specially- designed case studies

Consider Your EPP How is your EPP meeting the criteria of trustworthiness for socially-constructed data? Credibility – participant’s perspective Confirmability – corroboration by others Dependability – account for context Transferability – among contexts

Questions, critiques, & comments are welcome Courtney Glazer Director of Educator Preparation Cameron University Lawton, Oklahoma cglazer@cameron.edu