Dr. Chin and Dr. Nettelhorst Winter 2018 A Preliminary Psychometric Investigation: SmartEvals and Psychology Departmental Instrument Dr. Chin and Dr. Nettelhorst Winter 2018

Why Psychometrics? Validity is simply the property of an assessment tool that indicates the tool does what it says it does And if it does that, then test scores actually have meaning! Wrapped up in the concept of validity, is reliability When we administer a test, we would like to know how much of their scores reflects “truth” and how much reflects error

Methods for Estimating Validity Content Validity Criterion Related Validity Factorial Validity Exploratory Factor Analysis (EFA) Confirmatory Factor Analysis (CFA) Construct Validity

Methods for Estimating Reliability Test-Retest Reliability Alternate Forms of Reliability Internal Consistency Reliability Split Half Reliability Cronbach’s Coefficient Alpha Inter-Rater Reliability

Overall Analytical Strategy Departmental Instrument N = 76 Exploratory Factor Analysis (EFA) Internal Consistency Reliability (Cronbach’s Alpha) SmartEvals Instrument N = 57

Departmental Instrument Seventeen-item self-report measure Example: ”The instructor was enthusiastic about the subject matter” Likert-type scale Strongly Agree Agree Neither Agree nor Disagree Disagree Strongly Disagree

Departmental-EFA Model Specifications Conducted three standard EFAs with oblique geomin rotation. 1-factor, 2-factor, 3-factors, and 4-factors Conducted two bifactor EFAs with bi-geomin rotation One higher order factor and two subfactors One higher order factor and three subfactors

Departmental-EFA Model Specifications Used weighted least squares estimation (WLSMV) to model the data. Inspection of Eigenvalues Examined Global Fit Indices High CFI, TLI values (>.95), low RMSEA (<.06), and low SRMR (>.08) values were interpreted as indicative of strong model fit. Examined Factor Loadings > .30

Departmental-Bifactor model with two subfactors Refer to Handout for Pattern of Factor Loadings

Perceived Competency Student Assistance E1 E2 E3 E4 E5 E6 E7 E8 E9 E10   V1 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E17 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 Teaching Effectiveness  

Departmental-Bifactor model with two subfactors Reliability

Departmental-Bifactor model with two subfactors Reliability Total Score .92 Excellent) Factor 1 (Perceptions of Competency) .76 (Acceptable) Factor 2 (Assistance to Students) .83 (Good)

SmartEvals Instrument Ten-item self-report measure (excluding items 1 and 2) Example: ”The expectations of class were clearly communicated at the beginning of the course” Likert-type scale Strongly Agree Agree Neutral Disagree Strongly Disagree

SmartEvals-EFA Model Specifications Conducted two standard EFAs with oblique geomin rotation. 1-factor and 2-factor models Used weighted least squares estimation (WLSMV) to model the data. Inspection of Eigenvalues Examined Global Fit Indices High CFI, TLI values (>.95), low RMSEA (<.06), and low SRMR (>.08) values were interpreted as indicative of strong model fit. Examined Factor Loadings > .30

SmartEvals-One factor model Refer to Handout for Pattern of Factor Loadings

SmartEvals-Two factor model Refer to Handout for Pattern of Factor Loadings

Cronbach’s Alpha = .865 (Good) V1 E1 E2 E3   E4 E5 E6 E7 E8 E9 E10 V2 V3 V4 V5 V6 V7 V8 V9 V10 Teaching Effectiveness Cronbach’s Alpha = .865 (Good)

Implications & Future Research Preliminary evidence supports the use of total scores for SmartEvals and Departmental Instrument Larger sample size needed to determine generalizability of results Concerns over overfitting the data due to small sample size Other types of validity: Convergent Validity: Does the SmartEval function equally when administered in class versus online? Divergent Validity: Does the SmartEval total score distinguish between overall course quality and students’ grades in the course? Construct Validity: Can the SmartEvals detect changes in teaching effectiveness when instructors implement a novel teaching technique?

Thank you!