1. Susan Lowes, Ph.D. Devayani Tirthali, Ed.D. Peiyi Lin, Ed.D. Institute for Learning Technologies, Teachers College/Columbia University Selen Turkay, Ed.D. Harvard Initiative for Learning and Teaching, Harvard University Digital Pile Sorts: An Engaging Way to Assess Domain Understanding Skill Building Workshop American Evaluation Association Annual Meeting, October 17, 2013

2. Workshop format  Introduction to pile sorts (10 minutes)  What are pile sorts?  How do you do pile sorts?  Types of pile sorts  Three examples of pile sorts (15 minutes)  Design your own pile sort (30 minutes)  Share results (15 minutes)  Analyzing pile sort results (15 minutes)

3. Pile sorts as a eliciting technique

4. What are pile sorts?  Pile sorts, or card sorts, are an elicitation technique that is used to explore a domain or assess how participants understand a domain  Cards can have words, images, or both—or even be objects  Early use was by anthropologists to understand the way an unfamiliar culture conceptualizes a domain

5. How do you do a pile sort?  The user is given a stack of objects or cards with words and/or images and asked to sort them into piles that are similar or related  They may then be asked to explain their rationale.

6. Types of pile sorts

7. Open (unconstained) pile sorts  Examples:  Social class in the West Indies (Lowes 1994): Participants were given a set of cards with family names and asked to arrange them to represent “how things were socially.”  Social identity of adolescents in Puerto Rico (Giles et al., 1979): Participants were asked to sort dimensions of social identity into those that did and did not apply to them.  The participant determines how many piles and describes what they are

8. Closed (constrained) pile sorts  The number and/or names of the piles are pre- determined  Examples :  Gendered perceptions of food: Which of these are girls’ foods, boys’ foods, and both? (Roos 1998)  Expanding perceptions of engineers: Which of these cards represent engineers?  Understanding of the pervasive nature of sensors: Which objects have sensors?

9. Repeated pile sorts  The participants sort the cards in a number of different ways, as determined by the researcher  Example :  How do high school students perceive video games: (1) Which of these games do you play/not play? (2) What kinds of games are these? (3) What kind of person plays each game?

10. Successive pile sorts  Participants sort cards into piles, then sort one or more of the piles into sub-piles  Example:  Student perceptions of STEM characters in video games: How would you break these characters into two groups? For each group, which do you like most? Which do you not like?

11. Other types of pile sorts  Triadic pile sorts  For three items, select the two most alike  For three items, select the two most alike and the two most different  Pile sorts with rankings  After pile sort is complete, participants are asked to rank the piles

12. Engineer pile sort (constrained) Sensor pile sort (constrained with additional questions) Videogame character pile sort (unconstrained, successive, with additional questions) Three examples of pile sorts

13. Engineers: Which are engineers?

14. Sensors: Which objects have sensors?

15. Videogame characters: Step 1--sort

16. Step 2: Judgments about characters

17. Step 3: Perceptions of characters Example of a middle school girl’s pilesort  Doctor pile > Characters least liked >

18. Advantages and challenges

19. Advantages of pile sorts  Are fast and easy to administer; little burden on participants  Are interesting, even fun, for participants including children  Can elicit information that participants cannot articulate and/or is not easily accessible through interviews or surveys  May be a more reliable indicator of expertise than survey results  Can be used as pre-post assessments  Can be analyzed in multiple ways—content analysis, simple metrics, or using complex statistics

20. Challenges of pile sorts  Must know the domain, so free listing is often used first  Choosing the cards is tricky, needs extensive testing  Participants may not recognize objects  May focus on some words, ignoring the meaning  Unconstrained pile sorts are particularly difficult to analyze since every sort may be different--best used for exploring  With paper cards, difficult to capture the data or administer to large numbers of people  Digitization solves the last two of these challenges

21. Design your own pile sort Share your results Design your own pile sort

22. Exercise: Design your own pile sort  Pick your domain  What concepts are you getting at?  Assessing or exploring?  Constrained or unconstrained  Successive or one time  With explanation

23. Share results (15 minutes)

24. Analyzing pile sort results

25. Visualizing pre-to-post change Engineers Not Engineers Pre Post N= 15 teachers Using Excel worksheet to visualize pre-post change

26. Drag-and-drop (n=517)Open-ended (n=517) Difference between pre and post for drag-and-drop (left) and open-ended (right) Percent change, sort vs. explain

27. Detail for one item Hand dryer (drag-and-drop) 88% --> 95% Hand dryer (open-ended) 50% --> 63%

28. Similarity Matrix

29. True positive/negative Drag-and-dropOpen-ended

30. Plotting on x/y axis

31. MDS: Plotting multiple piles  Graphical representation of similarity relations between two cards shown as distances within a space.  Prepare sorting data by converting into a dissimilarity matrix

32. Acknowledgments  The engineer pile sort was developed as part of the evaluation of the Build-IT Scale-UP (BISU) project under an NSF ITEST scale-up grant to the Center for Innovation in Engineering and Science (CIESE) at Stevens Institute of Technology.  The sensor pile sort was developed as part of the evaluation of the SENSE IT project, an NSF ITEST strategies grant to Clarkson University.  The video game character pile sort was funded in part by Microsoft Research through the Games for Learning Institute.

33. For additional information  For information on the digital versions of these pile sorts, which were created in Flash and Javascript, contact, Susan Lowes (lowes@tc.edu)  For information on the individual pile sorts, contact:  Engineer pile sort: Devayani Tirthali (dgt2103@columbia.edu)  Character (games) pile sort: Selen Turkay (selenturkay17@gmail.com)  Sensor pile sort: Peiyi Lin (pl2151@columbia.edu)