1. Instrumentation

2. Instruments Questionnaires Surveys Interviews
How do you know what to ask?

3. measuring variables
“…a characteristic or attribute of an individual or an organization that can be measured or observed by the researcher and varies among individuals or organizations studied” (Chp 5, p.84)
Variables are measured using a measurement instrument, a tool, a data collection instrument or just an instrument

4. Levels of measurement Hierarchical
Determine what statistical analysis can be used
NOMINAL
ORDINAL
INTERVAL
RATIO
Lowest Level
Highest Level

5. NOMINAL (categorical)
Names, nomenclature, or labels used to classify
Two requirements:
Categories must be mutually exclusive
Categories have to be exhaustive
Nominal measures do not convey any value to what is measured …just name it (e.g. race, sex, class)
Tests of difference (nonparametric):
Chi-square
Measure of association (correlation):
Contingency coefficient

6. Ordinal (categorical)
Three requirements:
Categories have to be mutually exclusive
Categories must be exhaustive
Categories allow rank ordering
Categories represent relatively more or less of something, however the distance between categories cannot be measured
How would you describe your level of health?
Excellent
Good
Fair
Poor

7. Ordinal cont. Nonparametric statistics Correlational coefficients
Spearman r
Kendal r

8. Interval (continuous)
Category Rules:
Mutually exclusive
Exhaustive
Rank order
Widths of categories must be the same which allows for the distance between categories to be measured
No absolute zero (e.g. temperature)
Parametric statistics:
Means, standard deviations
Pearson correlations
t-test
F-test

9. Ratio (continuous) Category Rules: Sometimes referred to a numerical
Mutually exclusive
Exhaustive
Rank order
Widths of categories must be the same which allows for the distance between categories to be measured
Scale has an absolute zero (e.g. age, height, weight, time)
Sometimes referred to a numerical
All statistical tests

10. Psychometric properties
Accuracy = validity
Consistency = reliability
Fairness = appropriate for participants

11. Fairness Has traditionally been given the least attention
No one way to measure fairness
Is the instrument “fair” for individuals of various ethnic groups, educational levels, gender, etc.
Examples:
Cultural and language sensitivity
Assessment of conceptual and linguistic equivalence of other language versions of an instrument
Literacy
Evaluate the reading level of the instrument

12. Validity = accuracy
Does the instrument accurately measure what it is intending to measure?
Establishing Validity of an Instrument:
Criterion-related validity
Construct validity
Content validity
Face validity

13. Criterion-related validity
Data from a measurement instrument is correlated with data generated from a measure (criterion) of the concept being studied (usually an individual’s performance or behavior).
Use of a second measure of a concept as a criterion by which the validity of the new measure can be checked.
Two types:
Predictive validity – measure used will be correlated with a future performance or behavior
LSAT scores accurately predict success in law school in the future
SAT and ACT scores predict future college success

14. Criterion-related validity
Concurrent validity – a new instrument and an established (valid) instrument that measure the same thing are given to the same sample and the results of the new instrument correlate with the results of the established instrument
Beck’s Depression Inventory (established ) and the BDI II (new)
High positive correlation between scores is evidence of concurrent validity

15. Construct validity
If there is no existing instrument to compare to or the concept being measured is more abstract in nature, then construct validity is useful.
The degree to which a measure correlates with other measures it is theoretically expected to correlate with…(driven by theory)
Are measures positively or negatively associated with each other as would be expected by theory

16. Construct validity Two types:
Convergent - degree to which two measures which purport to be measuring the same topic positively correlate (converge)
Theory: Would we expect the relationship to be?
Instrument #1 measures person’s self-efficacy for regular exercise
Instrument #2 measures person’s exercise behavior
Discriminant – construct measured (self-efficacy for regular exercise) should not correlate with dissimilar variables (based on theory)
Measure of person’s self-efficacy for regular exercise would not be expected to positively correlate with a person’s inactivity

17. Content validity
Content validity is established during an instruments early development, not after completion.
Assessment of the correspondence between the items that make up an instrument and the content domain from which the items were selected
Step-by-step process

18. Face validity Weakest form of validity
Can be a good first step to establishing validity, but it should not replace other means for establishing validity
“On the face” the instrument appears to measure what it says it measures
Established by having individuals familiar with the concept look over the instrument to see if it appears to cover the concept it seeks to measure

19. Validity of screening and diagnostic tests
Sensitivity – ability of the test to identify correctly all screened individuals who actually have the disease/condition
a true positives
a + c true positives + false negatives
Specificity – the ability of the test to identify only nondiseased individuals who actually do not have the disease/condition
d true negatives
b + d false positives + true negatives

20. Validity is most important
If an instrument does not measure what it is supposed to (validity), then it does not matter if it is reliable.

21. Reliability
Consistency – extent to which a measure will produce the same or nearly the same results each time it is used
Reliability is estimated by computing a correlation coefficient (r) ..the closer the correlation coefficient is to 1.00, the greater the reliability
Values of r between .6 and .8 = moderate
Values of r above .8 = substantial correlation

22. Methods for determining reliability
Parallel Forms
Internal Consistency
Test-Retest
Rater
Interrator
intrarater

23. Parallel Forms Also called equivalent form or alternate form
Create different forms of the same measure that when given to the same participants will produce similar results (means, SD, item correlations)
Example: SAT and ACT exams

24. Internal consistency One of the most common methods used
Intercorrelations among individual items
Correlate each individual item and the total score
The greater the correlation, the higher the reliability
Statistical Measures:
Cronbach’s alpha
Kuder-Richardson(KR) 20 or 21 coefficient
Spearman-Bowman split-half reliability

25. Test-retest
Also called stability reliability as it provides evidence of stability over time
Same instrument, same group, same conditions, at two different points in time
Data from two administrations of the instrument are used to calculate a correlation coefficient
How much time should there be between administations?

26. Inter- and intra- rater reliability
Inter = Consistency of an observed event by different raters (e.g. three research team members are identifying themes from interview transcripts as part of a qualitative study)
Intra = consistent measurement or rating by the same person (e.g. I take blood pressure measures of community members)