1. Design Review “Paper Towels Assessment” Education 334X Technology-Based Student Achievement Assessment 2001.10.22 ‘Alim, Josie, Mike G, Bradley, Nina, Yunn Chyi, Johnnie, Jim

2. What does this assessment measure? Conceptual framework & locate assessment Conceptual analysis of –task –response format –scoring system What does prior research say? Observational Data: summary of notes from live observation and implications for simulation

3. In the conceptual framework Content knowledge Constrained Process knowledge Open Rich Lean Paper Towels

4. Conceptual Analysis DeclarativeProceduralSchematicStrategic Performance Task What is… weight, volume & relationship between them saturation area & length, purpose/use of different tools How to… read/write saturate control variables measure volume, weight etc. determine amount of water soaked up Why… saturate control variables use care in procedures recognize when has reached valid/reliable result recognize when unexpected factors affect result Response Format Which towel absorbs the most and which the least? Steps in procedureHow did you know? Completely wet? Same size? Scoring System Correct resultMethod for getting towel wet Determine result Care in procedure Saturation Determine result Care in procedure

5. For this type of performance assessment, prior research has found: Student notebook scores are exchangeable for direct observation scores. Notebook scores were slightly more reliable for students with prior experience with hands-on instruction. There is very high interrater reliability. Teachers can be trained to consistently judge performance. There is somewhat higher reliability for observed hands-on performance than for notebooks. There was moderate correlation between this type of assessment and the multiple choice test suggesting that different aspects of science achievement are being measured. There is low to moderate intertask reliability. This may be a concern because a number of tasks are needed in order to be able to generalize the performance to the domain. Sources: Baxter & Shavelson ’94- Evaluation of Procedure-Based Scoring for Hands-On Science Assessment Shavelson, Gao & Baxter ’93 - Sampling Variability of Performance Assessments

6. n = 1

7. Observation Consideration: Bias due to familiarity with test and scoring criteria. Single variable procedures only for method. Equalization primary concern for Keri. First Trial: Used dropper to saturate equal pieces. Counted drops and compared. Second Trial: Weighed equal pieces in scale. Third Trial: Used set amount of water in beaker to saturate. Compared amount of water left in beaker.

8. Valid for what kinds of instructional goals? To develop scientific reasoning, critical thinking, and inquiry/curiosity (3) (2) To design and conduct a sound scientific experiments, including identifying and controlling variables (develop procedural understanding) (4) To apply previously learned skills to new situations (6) To develop skills and appreciation for careful observation and measurement (5) “Paper Towels” is a Performance Assessment designed to support a hands-on, inquiry-based approach to learning and teaching that is essential in the primary grades. (1)

9. What is the utility of this hands-on performance assessment? Provides richer feedback regarding the student’s ability to solve problems using scientific reasoning The experience is instructional and non- intimidating Allows you to characterize students’ procedural strengths and weaknesses (good feedback to teacher) It’s much more collaborative and fun! More costly than multiple- choice, paper-pencil test (new and additional hands- on materials) Requires more time (cost) to score and be trained to score; requires trained scorers (harder to score) Requires more set-up time Requires more class time to execute the test The lab-book depends on literacy skills It’s weak in content (+) (-)

10. Reliability Inter-rater Very high,.94 (according to authors) Inter-task Not relevant, only one task is offered Sources of error Very few opportunities for discrepancy Limited to judgment calls: - care taken? - saturated?

11. What is the construct validity of PT assessment? Instructional GoalsWhat PT measures? 1.To develop reasoning, critical thinking and inquiry skills 1.Effective explanation of principles underlying the performance tasks. 2. Ability to provide a goal-directed, efficient strategies (problem solving) 3. Ability to self-monitor, constantly checking on their thinking and reasoning to determine the result. 2.To design and conduct a sound scientific experiment Ability to assess the nature of the problem and construct a plan (mental model) to carry out the strategy 4.To transfer their knowledgeProblem solving skills (since PT is content lean) 5.Develop skills and appreciation for careful observation and measurement Care in measuring &/or saturation

12. What is the construct validity of PT assessment? Content validity: The instructional goals of the lesson parallel with what PT is measuring (as shown in the table) Cognitive process PT assess students’ ability to explain the underlying principles, construct a plan (mental model) to initiate the strategy, devise strategies to solve the problem, monitor their thinking and reasoning… while PERFORMING the task (vs. recalling, reciting). These cognitive processes indicate effective learning of the subject matter (Baxtel & Elder, 1996). Performance level Research by Baxter et al (1992) indicates that PA measures correlated less with traditional ability than did a MC achievement test, ie: this PA doesn’t measure dlec Exchangeability for other methods (M-C, C-R, Teacher observation?) Research has shown performance assessment (e.g. PT) is exchangeable for direct observation. As a content lean performance assessment, PT measures different aspects of science achievement (problem solving skills) as Multiple Choice (MC) and Construct Response (CR), which concentrates more on domain knowledge. Hence they are not exchangeable.

13. Implications for Virtualizing PT Real world mistakes vs. virtual world mistakes - How many can/should we accommodate? Drop size, scale validity, scattered moisture, size of towels… How many roads (scenarios) can we account for? Cross-method results (combining results from multiple experiments) Can human observation really be replaced by computer tracking? Now that we’re not using the notebooks, should we bring back tracking “care” (eg: alarms or flags set for missing “care steps”) Do we still need to give them some hands-on materials? Can we really do this??