1. Windows Into The Mind Richard J. Shavelson & Maria Araceli Ruiz-Primo Stanford University Invited Talk University of Ancona Ancona, Italy June 26, 2000

2. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 2 DENNIS THE MENACE “GEE! I’M GLAD WE DID CONCEPT MAPS - NOW WE BOTH KNOW WHAT I LEARNED TODAY!!! ” Assessing Knowledge Structure

3. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 3 Overview Cognitive theory underlying assessments of knowledge structure Direct and indirect methods of assessing “cognitive structure” Direct method: Concept maps Indirect method: Cognitive maps Link between concept and cognitive maps Concluding comments

4. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 4 Toward An Achievement Framework: Knowledge Types and Dimensions Extent (How much?) Structure (How is it organized?) Others (Precision? Efficiency? Automaticity?) DeclarativeProceduralStrategic KnowledgeKnowledgeKnowledge (Knowing the “that”)(Knowing the “how”)(Knowing the “which,” “when,” and “why”) CognitiveTools:PlanningMonitoring Domain-specific content: facts concepts principles Domain-specific production systems Problem schemata/ strategies/ operation systems Proficiency LowHigh

5. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 5 is an example of Associative and Semantic Models of Memory Nodes represent concepts Lines represent relations among concepts Probe memory by asking about concepts (or lines) Associative Semantic is a contains

6. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 6 Concepts Maps Methods (Mapping Techniques) –Constructing A Map –Fill-In-The-Node Map –Fill-In-The-Line Map –Many Others Data structure and scoring methods depend on the technique

7. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 7 What Are Concept Maps? A Concept Map of a Concept Map! graph Concept Map consist of is a nodes labeled lines concepts relation denote a represent linked by between a pair of

8. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 8 Concept Map Assessment Components TaskTask that invites a student to provide evidence bearing on his or her declarative knowledge structure in a domain FormatresponseFormat for the student’s response Scoring systemScoring system by which the student’s concept map can be evaluated accurately and consistently

9. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 9 Are There Different Types of Concept Maps? Yes, more than we would like! Variations in the task, the response format, and the scoring system produce different concept mapping techniques.

10. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 10 Concept Map Assessment: An Example The following map shows the relationship among concepts related to genetic continuity. The concepts are arranged hierarchically and linked each other. Please examine the map and supply, in the space provided, a word or two for labeling each link such that association between concepts is made clear. Task Response Format Genetic continuity Asexual reproduction Sexual reproduction Meiosis Male gamete Female gamete 1. through 2. producer 3. requires 4. called 5. fuse into Scoring System One point is given for each correct relationship One point is given for each level up until two levels beyond the last branching One point is given for the first branching where two or more concepts are connected to the concept above. Three points are given for any subsequent branching One point is given for each cross link Source: Tamir, P. (1995). Science Assessment. In M. Birenbaum & F. J. R. C. Dochy (Ed.). Alternatives in assessment of achievements, learning processes, and prior knowledge. Boston: Kluwer Academic Publishers.12

11. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 11 Another Example Task Response Format Scoring System Construct a concept map that reflects what you know about Ions, Molecules, and Compounds. The map should have 10 concepts in it. Organize the terms in relation to one another in any way you want. Focuses on the accuracy of the propositions: Four points if the proposition is outstanding and complete Three points if the proposition is correct and complete Two points if the proposition is incomplete but correct One point if the proposition is correct but does not show understanding between the two concepts. Zero if the proposition Blank page for constructing the map

12. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 12 On The Reliability & Validity of Concept Maps ReliabilityReliability: Can a student’s concept maps be consistently scored by different persons? Equivalence of Mapping TechniquesEquivalence of Mapping Techniques: Do all mapping techniques provide the same picture of a student’s declarative knowledge structure? Task DemandsTask Demands: Do different mapping techniques impose different cognitive demands on a student? Comparing Concept Maps With Traditional AssessmentsComparing Concept Maps With Traditional Assessments: Do concept maps measure something different than multiple-choice tests?

13. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 13 Study 1: Tasks No-Concepts ProvidedConcepts Provided Construct a concept map about Ions, Molecules, and Compounds. Select another 7 concepts that you think are the most important in explaining the topic. Organize the concepts Construct a concept map about Ions, Molecules, and Compounds. Examine the 10 concepts and construct the map. Organize the concepts Redraw the map so someone else can read it. Mapping Technique 1 Mapping Technique 2 LIST OF CONCEPTS acids anions...

14. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 14 Response Format Blank page to construct the map Construct-a-Map: No Concepts Construct-a-Map: Concepts Provided Blank page to construct the map Concepts: Atoms, Compounds, Ions...

15. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 15 Scoring System Proposition Accuracy:Proposition Accuracy: Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent ConvergenceConvergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map Salience:Salience: Proportion of valid propositions to the total number of propositions in student’s map Based on a criterion map, scoring focuses on:

16. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 16 Proposition “Quality” Quality of Proposition Definition Outstanding proposition. Complete and correct. It shows a deep understanding of the relation between the two concepts. acids-compounds: < that gives off H+ when dissolved in water are Excellent 4 points Good Complete and correct proposition. It shows a good understanding of the relation between the two concepts. acids-compounds: > are examples of 3 points Poor Incomplete but correct proposition. It shows partial understanding of the relation between the two concepts. acids-compounds: < form 2 points Don’t Care Although valid, the proposition does not show understanding between the two concepts. acids-compounds: > is a different concept 1 points Invalid Incorrect proposition. acids-compounds: > made of 0 points

17. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 17 Generalizability(Reliability) of Scores Percent of Total Variability and G Coefficients for Proposition Validity Total Scores No-Concepts, Sample A, and B Sample A & B Person (p) Rater (r) Sample (s) p x r p x s r x c prc,e     2 71.6478.67.15 0 0 0.79 22.8117.64.01.18 5.37 2.69.89.88.78 2 (n r =2,n s =3) (n r =1,n s =1) (n r =2,n s =2)

18. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 18 Discriminant Validity Type of Score No-Concepts Sample ASample B Proposition Validity Congruence Salience Correlation Between Multiple-Choice Test and Concept Map Scores.58 a.45.64.66.61.63.55.50

19. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 19 Study 2: Tasks Construct-a-Map Fill-in-the-NodesFill-in-the-Linking Lines Construct a map. Use the 20 concepts provided to construct the map. Organize the concepts any way you want. Check the map. Redraw the map so someone else can read it. Examine the skeleton map, the blank nodes and the terms provided. Select the term that corresponds to each blank node. Write the term inside the circle. Check that all blank nodes are filled-in. Examine the skeleton map, the blank linking lines and the linking words provided. Select the linking words that correspond to each blank linking line. Write the linking words on the blank linking line. Check that all blank lines are filled-in.

20. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 20 Response Format Construct-a-Map Fill-in-the-Nodes atoms compounds N2O4 Elements, Atoms, and Compounds Fill-in-the-Linking Lines Elements, Atoms, and Compounds formed with cations and anions are formed from molecules sharing example of made up of opposite composed of that begins with H are lose or gain electrons to form with 3 or more elements are atoms compounds N2O4 made up of opposite composed of with 3 or more elements are ions ionic compounds molecular compounds ternary ionic acids formed from molecules sharing Blank page to construct the map Concepts: Elements, Atoms, Compounds...

21. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 21 Construct-A-Map Construct a concept map that reflects what you know about what a concept map is. Organize the terms in relation to one another in any way you want.

22. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 22 Fill-In-The-Node Examine the map, the blank nodes and the terms provided in the list below. Select the term that corresponds to each node and write it down inside the circle. Use each term only once.

23. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 23 Fill-In-The-Linking-Lines Examine the map, the blank lines and the linking words provided on the list below. Select the word(s) that corresponds to each line and write the number down in the line. Use each number only once.

24. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 24 Construct-a-MapFill-in-the-Map Scoring System :Proposition Accuracy: Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent Convergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map :Salience: Proportion of valid propositions to the total number of propositions in student’s map Correct/Incorrect: Student’s responses on the skeleton map were scored as correct or incorrect.

25. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 25 Are Fill-In-The-Map Scores Sensitive To The Nodes/Linking Lines Selected To Be Filled-In? Type of Skeleton Map Mean S.D. Means and Standard Deviations by Type of Map and Sample Fill-in-the-nodes Sample 18011.211.42 Sample 27210.801.74 Fill-in-the-linking lines Sample 1789.772.74 Sample 2738.993.09 (Max = 12) n

26. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 26 Are The Two Forms Of Fill-In- The-Map Equivalent? Sequence Fill-in-the-nodes Fill-in-the-linking lines Mean S.D. Means and Standard Deviations by Type of Map and Sequence 1 Nodes 1-Lines 14311.091.529.722.84 2 Nodes 1-Lines 23611.031.339.313.06 3 Nodes 2-Lines 13510.631.819.832.65 4 Nodes 2-Lines 23710.971.678.683.13 Total15211.021.599.362.93 n

27. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 27 Can The Two Mapping Techniques Be Considered Equivalent? Type of Assessment n Max Mean S.D. Means and Standard Deviations across the Three Types of Assessments Construct-A-Map Proposition Accuracy15213553.9122.17 Convergence1521.50.19 Salience1521.73.17 Fill-In Fill-in-the-nodes1521211.021.59 Fill-in-the-lines151129.392.93 Multiple-Choice Test 1503024.053.74

28. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 28 Do The Mapping Techniques Provide Information Similar To That Provided By A Multiple-Choice Test? Type of Assessment Construct-A-Map (C-M) (.99) Fill-in-the-nodes-NOD (FI-N).47 (.71) Fill-in-the-lines-LIN (FI-L).44.40 (.85) Multiple-Choice Test (MC).44.37.53(.74) C-MFI-NFI-LMC StructureExtent

29. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 29 Other Empirical Evidence Correlations tell you about rank ordering on different measures--that’s only part of the story Evidence is needed for the cognitive claims that different methods measure somewhat different aspects of achievement: –Talk aloud –Focus group –Group work –Other?

30. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 30 Talk Aloud with Concept Maps: Method Variation Verbal Codes Proportion Scores Correspondence Between Directedness and Inferred Activity Low Directed High Directed

31. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 31 Tentative Concept Maps Conclusions Students’ maps can be consistently scored by different persons even when complex judgments, such as proposition quality, are required Different mapping methods provide different pictures of students’ declarative knowledge structure. The construct-a-map method provides opportunities to reveal students’ conceptual understanding Different mapping techniques impose different cognitive demands on students. Highly structured mapping techniques allow students to respond by elimination or guessing, whereas constructed response do not. Correlations between concept-map and multiple-choice scores are positive and moderate suggesting that these two types of assessment measure overlapping but somewhat different aspects of declarative knowledge

32. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 32 Cognitive Maps Methods –Word Association –Similarity Judgment –Card Sorting –Tree Building Data structure: Proximities or distances Scaling methods: Networks or hierarchical clusters

33. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 33 Example: Similarity Judgments Rate the similarity of each pair of terms by circling one of the numbers provided or “DK -- Don’t Know 1.Central TendencyMean (Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK 2.HypothesisDescription (Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK

34. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 34 Concept x Concept Proximity Matrix For A Student

35. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 35 Variance Variability Description Mean Central TendencyModeMedianStatisticsInferenceHypothesis t-test ANOVA RegressionCorrelationPopulationSample Network Representation of Proximities

36. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 36 Small Pilot Study (N=28) Of Similarity Judgments And: Concept maps Multiple choice Terms: Variability, Variance, Standard Deviation Variability Variance Std. Dev. measures is the square root of 1. Which of the following refers to a procedure for making generalizations from a limited random sample of data? a. Statistical Inference b. Intuition c. Population Parameterization d. Standardization Source: Ed Wiley’s 2nd Year Project

37. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 37 Inference Sample Hypothesis Population ANOVA Linear Regression refers to relationships within the uses sample to test ideas in the is used to test ideas about population in is used to test hypotheses in is used to test population-related addresses ideas about variables in population in the process of is a procedure in statistical is one special case of the general linear model, along with is a procedure in statistical Sample Portion Of Concept Map

38. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 38 Correlation Among Methods Type of Assessment Pretest Concept Map.155 Similarity Judgment.554*.251 Posttest Concept Map.330 Similarity Judgment.434*.706* Multiple Choice Concept Map * p <.01

39. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 39 What If We Could? Collect cognitive structure information indirectly by computer Immediately show a student a representation of her knowledge structure Permit her to modify the structure And label the lines?

40. July 27, 2000Shavelson & Ruiz-Primo: Windows Into The Mind 40 Conclusions Concept and cognitive maps appear to tap a different aspect of declarative knowledge than do multiple-choice tests Concept and cognitive maps may provide similar representations of structure although this has not been tested directly By assessing different aspects of knowledge, we may broaden what gets taught!