Susan Lowes, Ph.D. Devayani Tirthali, Ed.D. Peiyi Lin, Ed.D. Institute for Learning Technologies, Teachers College/Columbia University Selen Turkay, Ed.D.

Slides:



Advertisements
Similar presentations
Performance Assessment
Advertisements

MMAP Middle School Math Through Applications Project Dahwun Deepak Gazi Scott Sun-Young.
Evaluating Student Achievement
DATA TRACKING AND EVALUATION 1. Goal of the STEP program: To increase the number of STEM graduates within the five-year period of the grant. You have.
Conceptual Graph Analysis Chapter 20 Lori Nuth | EDIT 730 | Fall 2005.
Best Practices & Strategies Beth Kuykendall January 5, 2008.
Nigeria/MEMS Brown Bag Presentation: Elicitation Techniques May 31,2006.
Sit for Autism Developed by The Center of Excellence on Autism Spectrum Disorders Southern Connecticut State University Ruth Eren, Ed.D. Director.
By Kreetha Kaewkhong The development of an inquiry-based learning unit in light reflection and refraction DEPARTMENT OF PHYSICS FACULTY OF SCIENCE MAHIDOL.
CAP 252 Lecture Topic: Requirement Analysis Class Exercise: Use Cases.
Does Schema-Based Instruction and Self-Monitoring Influence Seventh Grade Students’ Proportional Thinking? Asha Jitendra, University of Minnesota Jon R.
* Calculating with Candy * Curriculum for teaching Addition, Subtraction, and Estimation Curriculum for teaching Addition, Subtraction, and Estimation.
Improving Students’ Flexibility in Algebra: The Benefits of Comparison Jon R. Star Michigan State University (Harvard University, as of July 2007)
T4E Teaching Book Study Lunch and Learn January 13, 2011.
Evaluating Professional Development Debbie Junk, Coordinator for Mathematics Initiatives Mathematics Project Directors’ Meeting Tuesday, October 9th Austin,
Learning through Service: The Contribution of Service- Learning to First Year Pre-Service Teachers Miranda Lin, Ph.D., Alan Bates, Ph.D., & Ashley Olson.
The Computer Science Course at Omar Al-Mukhtar University, Libya The Computer Science Course at Omar Al-Mukhtar University, Libya User-Centered Design.
Women in Games Workshop Los Angeles, CA – October 20, 2011 Noelle Griffin, PhD Assistant Director for Research and Evaluation CRESST STEM, Gaming, and.
Student Centered Learning
Evaluating NSF Programs
Developing an Effective Evaluation to Check for Understanding Susan E. Schultz, Ph.D. Evaluation Consultant PARK Teachers.
Business and Management Research
Brown, Q., Mongan, W., Garbarine, E., Kusic, D., Fromm, E., & Fontecchio, A. (2008). Computer Aided Instruction as a Vehicle for Problem Solving: Scratch.
Bettina Matysiak PIDP 3104November Introduction  What are “application cards”?  an informal assessment technique which allows instructors to.
Making Clickers Work for You Dr. Stephanie V. Chasteen & Dr. Steven Pollock Workshop developed.
PRESCHOOL LITERACY A Seminar for Preschool Teachers About Dialogic Reading Presented By: Darlene Shank.
Robotics Academy Bringing robotics to the younger crowd.
John Seelke University of Maryland College Park Preparing and Supporting Candidates for the edTPA 1.
 “…the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and.
Understanding of the fundamentals ◦ By the end of the class, 70% of the students will be able to:  Correctly draw freebody diagrams of 2D truss structures.
EVALUATION RESEARCH. Evaluation Research  How do we begin?  What are the different types of evaluation research?  How do these different types fit.
Performance-Based Assessment Authentic Assessment
8 th Grade Integers Natalie Menuau EDU Prof. R. Moroney Summer 2010.
Anchor Standards – A Way for Art Sandy Roe Nanette Nichols WDMESC.
Introduction to Kentucky’s Professional Growth and Effectiveness System and Framework for Teaching.
Using Common Core State Standards of Seventh Grade Mathematics in the Application of NXT LEGO® Robotics for CReSIS Middle School Students.
Middle School Students’ Technology Practices and Preferences: Re-Examining Gender Differences Miller, L. D., Schweingruber, H., & Brandenburg, C. L. (2001).
Jim Dorward Sarah Giersch Kaye Howe Rena Janke Mimi Recker Andy Walker NSF Awards: NSDL ;TPC Using Online Science & Math Resources in Classrooms.
The Intruiging World of Educational Game Sites & STEM Presenters: Jackie McCarthy Trevor Rice.
Integrate English as a Second Language (ESL) strategies into the teacher preparation program for all students the semester prior to student teaching. Require.
IEARN A “How To” Manual. Implementation Three easy steps to Implementation 1. Pick a Project 2. Use your imagination 3. Go to it!
111 MakingConnections Focus on Assessment. 222 Facilitator/s: Date:
Writing Qualitative Research Purpose and Questions A Cheat Sheet
SciGirls Tallahassee Evaluation and Research Roxanne Hughes, PhD (Director of the Center for Integrating Research and Learning) Presentation at SciGirls.
M= Math in STEM College and Career Ready- Conference Summer, 2015.
IST_Seminar II CHAPTER 12 Instructional Methods. Objectives: Students will: Explain the role of all teachers in the development of critical thinking skills.
Developing an Effective Evaluation to Check for Understanding Part 2 Susan E. Schultz, Ph.D. Evaluation Consultant PARK Teachers.
Sandi Simpkins, Ph.D. Do Families Matter for Adolescents’ Pursuit of STEM?
MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings.
The 5 E Instructional Model. Concepts of Science Science Content Process Skills of Science.
Introduction to Modeling in math. What is Modeling? Modeling is a strategy to represent the important structures of problems so they can more easily be.
Teaching with Data: Context and Resources Sean Fox, SERC Carleton College.
PREPARATION OF QUESTIONNAIRES PREPARATION OF QUESTIONNAIRES Chapter - 4 Dr. BALAMURUGAN MUTHURAMAN
1 Far West Teacher Center Network - NYS Teaching Standards: Your Path to Highly Effective Teaching 2013 Far West Teacher Center Network Teaching is the.
13 strategies to use Powerpoint to support active learning in classroom.
Engineering is Elementary Inservice PD Program Christine M. Cunningham Museum of Science, Boston.
NEPF-ALIGNED STUDENT PERCEPTION SURVEY IMPLEMENTATION By Ms. Amanda Byrd.
Why iPracticeMath? “No Child Left Behind” is a nationwide standard that is enforced in every school system. With the new emphasis on common core mathematical.
Are students the only things that experience stress and strain? No, it’s a common phenomenon that is a key for engineers to understand materials that we.
Property Fitting Analysis
7.3 Steal the Pile (Card Game)
Simple Strategies for Monitoring Student Growth: Validate yourself!!
Animals use their senses for survival
A Web-based Approach to Enhance Preventive Medicine Education Outcomes for Third Year Medical Students 35th Annual Predoctoral Education Conference 2009.
Katherine M. Hyland, PhD Marieke Kruidering-Hall, PhD
Changes to the Educator Evaluation System
Helping English Learners Be Successful!
Technological Resources
Decimals: Connections to the Common Core and the IES Practice Guide
Presentation transcript:

Susan Lowes, Ph.D. Devayani Tirthali, Ed.D. Peiyi Lin, Ed.D. Institute for Learning Technologies, Teachers College/Columbia University Selen Turkay, Ed.D. Harvard Initiative for Learning and Teaching, Harvard University Digital Pile Sorts: An Engaging Way to Assess Domain Understanding Skill Building Workshop American Evaluation Association Annual Meeting, October 17, 2013

Workshop format  Introduction to pile sorts (10 minutes)  What are pile sorts?  How do you do pile sorts?  Types of pile sorts  Three examples of pile sorts (15 minutes)  Design your own pile sort (30 minutes)  Share results (15 minutes)  Analyzing pile sort results (15 minutes)

Pile sorts as a eliciting technique

What are pile sorts?  Pile sorts, or card sorts, are an elicitation technique that is used to explore a domain or assess how participants understand a domain  Cards can have words, images, or both—or even be objects  Early use was by anthropologists to understand the way an unfamiliar culture conceptualizes a domain

How do you do a pile sort?  The user is given a stack of objects or cards with words and/or images and asked to sort them into piles that are similar or related  They may then be asked to explain their rationale.

Types of pile sorts

Open (unconstained) pile sorts  Examples:  Social class in the West Indies (Lowes 1994): Participants were given a set of cards with family names and asked to arrange them to represent “how things were socially.”  Social identity of adolescents in Puerto Rico (Giles et al., 1979): Participants were asked to sort dimensions of social identity into those that did and did not apply to them.  The participant determines how many piles and describes what they are

Closed (constrained) pile sorts  The number and/or names of the piles are pre- determined  Examples :  Gendered perceptions of food: Which of these are girls’ foods, boys’ foods, and both? (Roos 1998)  Expanding perceptions of engineers: Which of these cards represent engineers?  Understanding of the pervasive nature of sensors: Which objects have sensors?

Repeated pile sorts  The participants sort the cards in a number of different ways, as determined by the researcher  Example :  How do high school students perceive video games: (1) Which of these games do you play/not play? (2) What kinds of games are these? (3) What kind of person plays each game?

Successive pile sorts  Participants sort cards into piles, then sort one or more of the piles into sub-piles  Example:  Student perceptions of STEM characters in video games: How would you break these characters into two groups? For each group, which do you like most? Which do you not like?

Other types of pile sorts  Triadic pile sorts  For three items, select the two most alike  For three items, select the two most alike and the two most different  Pile sorts with rankings  After pile sort is complete, participants are asked to rank the piles

Engineer pile sort (constrained) Sensor pile sort (constrained with additional questions) Videogame character pile sort (unconstrained, successive, with additional questions) Three examples of pile sorts

Engineers: Which are engineers?

Sensors: Which objects have sensors?

Videogame characters: Step 1--sort

Step 2: Judgments about characters

Step 3: Perceptions of characters Example of a middle school girl’s pilesort  Doctor pile > Characters least liked >

Advantages and challenges

Advantages of pile sorts  Are fast and easy to administer; little burden on participants  Are interesting, even fun, for participants including children  Can elicit information that participants cannot articulate and/or is not easily accessible through interviews or surveys  May be a more reliable indicator of expertise than survey results  Can be used as pre-post assessments  Can be analyzed in multiple ways—content analysis, simple metrics, or using complex statistics

Challenges of pile sorts  Must know the domain, so free listing is often used first  Choosing the cards is tricky, needs extensive testing  Participants may not recognize objects  May focus on some words, ignoring the meaning  Unconstrained pile sorts are particularly difficult to analyze since every sort may be different--best used for exploring  With paper cards, difficult to capture the data or administer to large numbers of people  Digitization solves the last two of these challenges

Design your own pile sort Share your results Design your own pile sort

Exercise: Design your own pile sort  Pick your domain  What concepts are you getting at?  Assessing or exploring?  Constrained or unconstrained  Successive or one time  With explanation

Share results (15 minutes)

Analyzing pile sort results

Visualizing pre-to-post change Engineers Not Engineers Pre Post N= 15 teachers Using Excel worksheet to visualize pre-post change

Drag-and-drop (n=517)Open-ended (n=517) Difference between pre and post for drag-and-drop (left) and open-ended (right) Percent change, sort vs. explain

Detail for one item Hand dryer (drag-and-drop) 88% --> 95% Hand dryer (open-ended) 50% --> 63%

Similarity Matrix

True positive/negative Drag-and-dropOpen-ended

Plotting on x/y axis

MDS: Plotting multiple piles  Graphical representation of similarity relations between two cards shown as distances within a space.  Prepare sorting data by converting into a dissimilarity matrix

Acknowledgments  The engineer pile sort was developed as part of the evaluation of the Build-IT Scale-UP (BISU) project under an NSF ITEST scale-up grant to the Center for Innovation in Engineering and Science (CIESE) at Stevens Institute of Technology.  The sensor pile sort was developed as part of the evaluation of the SENSE IT project, an NSF ITEST strategies grant to Clarkson University.  The video game character pile sort was funded in part by Microsoft Research through the Games for Learning Institute.

For additional information  For information on the digital versions of these pile sorts, which were created in Flash and Javascript, contact, Susan Lowes  For information on the individual pile sorts, contact:  Engineer pile sort: Devayani Tirthali  Character (games) pile sort: Selen Turkay  Sensor pile sort: Peiyi Lin