1. Dr. Chin and Dr. Nettelhorst Winter 2018
A Preliminary Psychometric Investigation: SmartEvals and Psychology Departmental Instrument
Dr. Chin and Dr. Nettelhorst
Winter 2018

2. Why Psychometrics?
Validity is simply the property of an assessment tool that indicates the tool does what it says it does
And if it does that, then test scores actually have meaning!
Wrapped up in the concept of validity, is reliability
When we administer a test, we would like to know how much of their scores reflects “truth” and how much reflects error

4. Methods for Estimating Validity
Content Validity
Criterion Related Validity
Factorial Validity
Exploratory Factor Analysis (EFA)
Confirmatory Factor Analysis (CFA)
Construct Validity

5. Methods for Estimating Reliability
Test-Retest Reliability
Alternate Forms of Reliability
Internal Consistency Reliability
Split Half Reliability
Cronbach’s Coefficient Alpha
Inter-Rater Reliability

6. Overall Analytical Strategy
Departmental Instrument
N = 76
Exploratory Factor Analysis (EFA)
Internal Consistency Reliability (Cronbach’s Alpha)
SmartEvals Instrument
N = 57

7. Departmental Instrument
Seventeen-item self-report measure
Example: ”The instructor was enthusiastic about the subject matter”
Likert-type scale
Strongly Agree
Agree
Neither Agree nor Disagree
Disagree
Strongly Disagree

8. Departmental-EFA Model Specifications
Conducted three standard EFAs with oblique geomin rotation.
1-factor, 2-factor, 3-factors, and 4-factors
Conducted two bifactor EFAs with bi-geomin rotation
One higher order factor and two subfactors
One higher order factor and three subfactors

9. Departmental-EFA Model Specifications
Used weighted least squares estimation (WLSMV) to model the data.
Inspection of Eigenvalues
Examined Global Fit Indices
High CFI, TLI values (>.95), low RMSEA (<.06), and low SRMR (>.08) values were interpreted as indicative of strong model fit.
Examined Factor Loadings
> .30

10. Departmental-Bifactor model with two subfactors
Refer to Handout for Pattern of Factor Loadings

11. Perceived Competency Student Assistance E1 E2 E3 E4 E5 E6 E7 E8 E9 E10V1 E1 E2 E3 E4 E5 E6 E7 E8 E9
E10
E11
E12
E13
E14
E15
E16
E17 V2 V3 V4 V5 V6 V7 V8 V9
V10
V11
V12
V13
V14
V15
V16
V17
Teaching
Effectiveness

12. Departmental-Bifactor model with two subfactors
Reliability

13. Departmental-Bifactor model with two subfactors
Reliability
Total Score
.92 Excellent)
Factor 1 (Perceptions of Competency)
.76 (Acceptable)
Factor 2 (Assistance to Students)
.83 (Good)

14. SmartEvals Instrument
Ten-item self-report measure (excluding items 1 and 2)
Example: ”The expectations of class were clearly communicated at the beginning of the course”
Likert-type scale
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

15. SmartEvals-EFA Model Specifications
Conducted two standard EFAs with oblique geomin rotation.
1-factor and 2-factor models
Used weighted least squares estimation (WLSMV) to model the data.
Inspection of Eigenvalues
Examined Global Fit Indices
High CFI, TLI values (>.95), low RMSEA (<.06), and low SRMR (>.08) values were interpreted as indicative of strong model fit.
Examined Factor Loadings
> .30

16. SmartEvals-One factor model
Refer to Handout for Pattern of Factor Loadings

17. SmartEvals-Two factor model
Refer to Handout for Pattern of Factor Loadings

18. Cronbach’s Alpha = .865 (Good)V1 E1 E2 E3 E4 E5 E6 E7 E8 E9
E10 V2 V3 V4 V5 V6 V7 V8 V9
V10
Teaching
Effectiveness
Cronbach’s Alpha = .865 (Good)

19. Implications & Future Research
Preliminary evidence supports the use of total scores for SmartEvals and Departmental Instrument
Larger sample size needed to determine generalizability of results
Concerns over overfitting the data due to small sample size
Other types of validity:
Convergent Validity: Does the SmartEval function equally when administered in class versus online?
Divergent Validity: Does the SmartEval total score distinguish between overall course quality and students’ grades in the course?
Construct Validity: Can the SmartEvals detect changes in teaching effectiveness when instructors implement a novel teaching technique?

20. Thank you!