United States Military Academy Duty, Honor, Country 1 Assessment, Evaluation, & Statistics: A Brief Overview WISTPC 2014 Florida International University Miami, FL Steven J. Condly, PhD United States Military Academy at West Point

United States Military Academy Duty, Honor, Country 2 Definitions Measurement (What do you know?) –Assigning numbers to things, events, people, actions, etc. Assessment (How do you know?) –Measurements, actions, processes, data that answer the question. Evaluation (How are we doing?) –Comparing results and observations with goals and objectives (implied or otherwise).

United States Military Academy Duty, Honor, Country 3 Implications Measurement (assigning #s) –Need for consistency, accuracy, and precision. Assessment (collected data) –Should relate to what it is purportedly describing. Evaluation (comparing results to standards) –Not entirely objective, but should be reasonable and logical.

United States Military Academy Duty, Honor, Country 4 Types of Evaluation Formative –Serves to strengthen or improve the program being evaluated –In-process Summative –Examines the effect(s) program –Examines how well program goals and objectives were met –Contains implications for corrective action

United States Military Academy Duty, Honor, Country 5 Formative Evaluation How well identified and defined is the problem? How well does the program deal with the problem? How well does the program progress? [feedback loop]

United States Military Academy Duty, Honor, Country 6 Summative Evaluation Similar to Formative, but looks back over the entire life of the program. Reaches conclusions regarding effectiveness, cost, overall success, and likelihood of generalizability (or moving on to the next level). Easier to perform if formative evaluations are being performed and data/results collected.

United States Military Academy Duty, Honor, Country 7 Types of Assessment Assessment is data collection with a purpose Really only two ways to do it: –Question –Observe Raw data have to be processed (statistics)

United States Military Academy Duty, Honor, Country 8 Statistics Select a good comparison criterion or group Standard statistical techniques are alright for Likert-scaled survey data Don’t use p-values (NHST) –Strongly influenced by sample size Small p does not necessarily indicate a stronger relationship or effect, or practical significance –What people think it is: P(H 0 =0|sample) –What it actually is: P(sample|H 0 =0) –How much there is there?

United States Military Academy Duty, Honor, Country 9 Effect Size Statistics For Likert or interval-level data, when comparing two groups, use Cohen’s d – M 1 - M 2 / [(s 1 + s 2 ) / 2] For ordinal data, when comparing two groups, use Probability of Superiority – MWU / (n 1 n 2 ) For correlations between two groups, use r 2 – (r) (r) x 100 gives % of variance explained

United States Military Academy Duty, Honor, Country 10 Websites Grissom, R. J. (1994). The probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79(2),