1. CS1 Assessment Using Memory Diagrams Mark A. Holliday David R. Luginbuhl Dept of Mathematics and Computer Science Western Carolina University SIGCSE Technical Symposium on Computer Science Education Norfolk, VA March 5, 2004

2. 10/21/2003 2 Overview  Background  Introduction to Memory Diagrams – A simple example  Qualitative Assessment – Noting and correcting common errors  Quantitative Assessment – Results so far  Conclusion

3. 10/21/2003 3 Motivation  Problem: – How to introduce and reinforce object-oriented concepts in an introductory CS course – How to help CS1 students visualize the state of the computer as the Java program executes

4. 10/21/2003 4 Motivation  Solution: – Develop a visual representation of each change in the state of memory (abstractly) due to the object- oriented effects of executing a sequence of Java statements – Memory diagrams are more detailed than the code since there is at least one change in the state of memory per statement. – Often there are several changes in the state of memory for a single statement.

5. 10/21/2003 5 Motivation  Solution continued: – Write the fragment in both Java and the visual language – The visual language must be detailed and precise – The visual representation accesses an alternative and supplemental mode of thinking

6. 10/21/2003 6 Relationship to Psychology  In other application domains psychologists have studied the effect on text comprehension by supplementing with visual representations – P. Goolkasian, Picture-Word Differences in a Sentence Verification Task, Memory and Cognition, 1996. – B. Henderson, personal communication. 2003.

7. 10/21/2003 7 An “Elementary” Analogy  In elementary school, many of us were taught the rules of good grammar through sentence diagramming – The teacher would show students the parts of a sentence with diagrams – Students would also be required to diagram sentences to better understand the concepts of grammar The white rabbit eats green lettuce. rabbit eats lettuce. The white green

8. 10/21/2003 8 Method Invocation otherColor = “red”; aColor = “blue”; // a) aColor = aColor.concat(otherColor); // rhs call is b) // rhs return is c) // lhs is d) String “blue” String aColor String otherColor String “red” concat ( ) String “bluered” concat X

9. 10/21/2003 9 Language Design Principles  Use shape to reinforce when concepts are the same – e.g., all kinds of variables (local variables, parameters, and fields) are rectangles  Use shape to reinforce when concepts differ – e.g., variables, objects, classes all have different shapes (rectangles, circles, and diamonds)  Use position to reinforce concepts – e.g., the reference contained in a reference variable starts inside the variable, not on the border – e.g., the rectangle representing an instance variable is placed inside the circle representing the object

10. 10/21/2003 10 Student Exposure to Diagrams  We introduce these diagrams to students on the first day of CS1 class  We ask students to produce diagrams of their own – In in-class groupwork – In weekly closed labs – On quizzes and tests

11. 10/21/2003 11 Student Exposure to Diagrams Example: in-class groupwork – all groups at blackboards so that everyone can see the diagrams and multiple group members can be drawing simultaneously – everyone must be able to explain during the demonstration – no chalk for the strongest group member – at least a third of “lecture” time is spent doing this (with an upperclass student helper assisting)

12. 10/21/2003 12 Student Exposure to Diagrams  How do we have the time for all the groupwork and diagramming?  Ours is a minimalist CS1: – no exception handling – no inheritance, abstract classes, or interfaces – no recursion – no inner classes or event-driven code – no graphics – but we do real console I/O (not hidden)  Bottom line: reinforcement of concepts in a number of contexts – But not just for learning…

13. 10/21/2003 13 Student Assessment  By having students use diagrams themselves, we have them demonstrate their comprehension of object- oriented concepts  We believe these diagrams have potential for measuring programming comprehension

14. 10/21/2003 14 Qualitative Student Assessment  Test problem from last year: – Diagram the program fragment at the right  Note last line particularly – Static public variable (System.out) – Method composition – PrintStream object – Dog object – Creation of a String object Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d) Student attempts to produce figure a

15. 10/21/2003 15 Common Mistakes  Figure a – creating the object as well as the variable – putting variable name inside variable rectangle Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d) Dog spot Solution: Student attempts to produce figure b

16. 10/21/2003 16 Common Mistakes  Figure b – not creating an object: putting “spot” inside spot rectangle – name field problems: not showing field’s rectangle putting field’s rectangle on object boundary not showing String object for “spot” and that the name field holds a reference to that String object – reference problems: arrow in wrong direction arrow starts on the boundary of the rectangle instead of inside the rectangle Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d)

17. 10/21/2003 17 Common Mistakes  Figure b – already placing the reference in the spot variable Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d) Dog spot Dog String “spot” name Solution: Student attempts to produce figure c

18. 10/21/2003 18 Common Mistakes  Figure c – placing the reference on the boundary of the spot rectangle instead of inside Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d) Dog spot Dog String “spot” name Solution: Student attempts to produce figure d

19. 10/21/2003 19 Common Mistakes  Figure d Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d) PrintStream out name () Dog Dog spot System PrintStream String “spot” println toString Solution: () String “name=spot”

20. 10/21/2003 20 Common Mistakes  Figure d – not show toString method invocation correctly – not show the String object created correctly – not show the reference to the new String object being passed as the argument to the println method call – not show that System.out is a static public field of the System class – now show println method invocation correctly Dog spot;// fig a) spot = new Dog("spot"); // right hand side is fig b) // left hand side and equal sign is fig c) System.out.println(spot.toString()); // fig d)

21. 10/21/2003 21 Quantitative Student Assessment  Previous example illustrates how the diagrams are used qualitatively to assess student comprehension – to pinpoint problems and provide proper reinforcement of specific concepts  Can we also use them for quantitative assessment?  Can we also evaluate how effective they are in assessment?

22. 10/21/2003 22 Quantitative Assessment (cont.)  Completed Study: Correlation of score on memory diagram question with rest of test and with course score  Three test questions from two CS1 classes  Two studies (correlation with rest of test and correlation with course score) for each of the three questions (three experiments)

23. 10/21/2003 23 Quantitative Assessment (cont.) An example quantitative comparison using the third test question (experiment three)  Histogram is sorted by memory diagram question score  The three series of scores appear to track each other  Mean score of the memory diagram question is significantly lower – not surprising since a correct memory diagram requires a deeper understanding of the effect of a program fragment

24. 10/21/2003 24 Conclusion – Memory Diagrams  A relatively low-tech approach for teaching OO concepts – Well-suited for classroom, labs, exams – “Write” a program fragment in both Java and in the visual language – Importance of shape and placement for reinforcing concepts – Having students make their own diagrams adds to this reinforcement

25. 10/21/2003 25 Conclusion – Memory Diagrams  Promise of diagrams for measuring comprehension – If students can diagram what is happening in memory, they are probably understanding the deeper meaning of the program – Measured assessment effectiveness relative to rest of test and course score – Currently measuring assessment effectiveness compared to corresponding Java program fragment

26. 10/21/2003 26 Conclusion – Memory Diagrams To learn more  Mark A. Holliday and David Luginbuhl, “Using Memory Diagrams When Teaching a Java-Based CS1”, Proc. of the 41st ACM Southeast Conference, Savannah, GA, March 2003.  Mark A. Holliday and David Luginbuhl, “CS1 Assessment Using Memory Diagrams”, Proc. Of SIGCSE 2004, Norfolk, VA, March, 2004  Mark A. Holliday, “Introducing Java Using Memory Diagrams”, (include short guide to the notation); http://cs.wcu.edu/~holliday/LectureNotes/150/lectures.html http://cs.wcu.edu/~holliday/LectureNotes/150/lectures.html  Presentation slides: http://cs.wcu.edu/~holliday/sigcse04.ppt