1. Internet based assessments via collaborative problem solving: The Assessment and Teaching of 21 st Century Skills (ATC21S™) Project Patrick Griffin Executive Director, ATC21S

2. Patrick Griffin, University of Melbourne, Executive DirectorATC21S Michael Stevenson, Cisco Vice President - Global Education, Former ATC21S Board Chair (2009–2010) Shelly Esque, Intel Vice President - Legal and Corporate Affairs, Current ATC21S Board Chair (2011–2012) Anthony Salcito, Microsoft Vice President - Education, Former ATC21S Board Chair (2010–2011) Esther Care, University of Melbourne, ATC21S International Research Coordinator Ministerial representatives from… –Australia: –Finland: –Singapore –USA: –Costa Rica: –Netherlands: EXECUTIVE BOARD PROJECT GOVERNANCE

3. ADVISORY BOARD Patrick Griffin, University of Melbourne, Executive Director ATC21S (Chair) Andreas Schleicher, Organization for Economic Co-operation and Development (OECD) Seamus Hegarty, International Association for the Evaluation of Educational Achievement (IEA) Irina Bokova (Director General UNESCO) Ray Adams, Technical Director PISA 2003-2012 Marc Durando, European Schoolnet Esther Care, University of Melbourne, ATC21S International Research Coordinator Stuart Elliott, National Academy of Sciences David Forster, International Testing Commission Robin Horn, World Bank Eugenio Eduardo Severin, Inter-American Development Bank National project managers from each founder country and associate country Task force member from each company: Katrina Reynan, Director, Cisco Martina Roth, Director, Intel Greg Butler, Director, Microsoft

4.  Traditional assessments may not be suited to measure many 21 st century skills  Goal is to develop new assessment approaches matched to new C21 skills and to advise systems, schools and teachers on the use of assessment data to help students develop higher order performances Focus: the Assessment and Teaching of 21 st Century Skills Phase 1 Conceptualise C21 skills and education output needs Phase 2 Skill Identification and hypotheses Phase 3 Development and coding via coglabs Phase 4 Pilot studies and trials Phase 5 Dissemination scale and policy

5. CONCEPTUALIZING THE SKILLS Assemble experts Assemble experts Define methods Define methods Explore practical and technical needs in the classroom Explore practical and technical needs in the classroom Create a new framework Create a new framework Five working groups were established:  Defining 21st-century skills: Ms. Senta Raizen, WestEd  Methodological issues: Dr. Mark Wilson, University of California, Berkeley  Technological issues: Dr. Beno Csapo, University of Szeged, Hungary  Classrooms and formative evaluation: Dr. John Bransford, University of Washington, and Dr. Marlene Scardamalia, University of Toronto  Policy frameworks and new assessments: Dr. Linda Darling-Hammond, Stanford University Five working groups were established:  Defining 21st-century skills: Ms. Senta Raizen, WestEd  Methodological issues: Dr. Mark Wilson, University of California, Berkeley  Technological issues: Dr. Beno Csapo, University of Szeged, Hungary  Classrooms and formative evaluation: Dr. John Bransford, University of Washington, and Dr. Marlene Scardamalia, University of Toronto  Policy frameworks and new assessments: Dr. Linda Darling-Hammond, Stanford University

6. 21ST-CENTURY SKILLS DEFINED ATC21SATC21S started by internationally defining 21st-century skills as four broad categories. ATC21S

9. Collaborative problem-solving ATC21S Assessment and Teaching of 21st Century Skills The University of Melbourne Cisco Intel Microsoft www.atc21s.org

10. Components Collaborative problem solving Social skillsParticipation Perspective taking Social regulation Cognitive skillsTask regulation Knowledge building

12. Participation skills over levels

13. Perspective taking skills

14. Social regulation skills

15. Task regulation skills

16. Knowledge building skills

18. Developin assessment Tasks for ATC21S Founder countries Associate and founder countries Draft -- Concept check for reality Panel – cognitive laboratory for codes Pilot – for administration Trials – for calibration

20. Tasks and Assignments to assess ICT Literacy Poetry (Graphic Organizers, Creating and Listening to Audio and video) Arctic Trek (Collaborative Notebook, Information Foraging)

22. ATC21S Twenty first century skills

23. Collecting the Data

24. Coding and scoring

27. Cognitive Behaviour U2L001A = presence of chat before any action Social Behaviour U2L00 4A = All positions hav been covered (providing player had access to 3 balls) U2L006A = sewuential placement of balls -6 combinations LMRLMR RMPRML RMLLMR LMRRML LLMMRR RRMMLL

29. Calibration and Interpretation

32. ElementAmplificationLowMiddleHigh Participation 1.Action Activity within environment No or very little activity in environment Activity in scaffolded context Activity in scaffolded and unscaffolded contexts 2.Interaction Interacting with, prompting and responding to the contributions of others Acknowledges communication Responding to cues in communication Initiating and promoting interaction or activity 3.Task completion Undertaking and completing a task or part of a task individually Maintains presence only (lurking) Identifies and attempts the task Perseveres in task as indicated by repeated attempts or multiple strategies Perspective taking 4.Responding adaptive responsiveness Ignoring, accepting or adapting contributions of others Contributions or prompts from others are ignored Contributions or prompts from others are taken into account Contributions or prompts of others are adapted and incorporated appropriately 5.Audience awareness Mutual modelling Awareness of how to adapt behaviour to increase suitability for others Contributions are not tailored to participants Contributions are modified for recipient understanding in the light of feedback Contributions are tailored to recipients (audience design) Social regulation 6.Negotiation Achieving a resolution or reaching compromise No attempts to negotiate joint understanding Comments on differences in perspective Negotiates through differences in perspective 7.Metamemory Self concept Recognising own strengths and weaknesses Notes own performance Comments on own performance in terms of appropriateness or adequacy Comments on own performance in terms of appropriateness or adequacy in the context of the task 8.Transactive memory Recognising strengths and weaknesses of others Notes performance of others Comments on performance of others in terms of appropriateness or adequacy Comments on performance of others in terms of appropriateness or adequacy in the context of the task 9.responsibility Initiative Assuming responsibility for ensuring aspects of task are completed by the group Undertakes activities largely independently of others Completes activities and reports to others Assumes group responsibility as indicated by use of second person plural or accepting others’ contribution 11. Resource managementManaging resources or people to complete a task Uses resources (or directs people) without consultation. Suggests that people or resources be used in part of a task Allocates people or resources to a task through to completion

33. ElementAmplificationLowMiddleHigh Knowledge Building 20. Knowledge acquisitionFollow path to gain knowledge Acquires knowledge as a result of being given it directly Deliberate single actions to acquire knowledge Knowledge acquired through multiple purposeful actions 12. Rules “If …then” (planning and executing) Formulating a course of action to address a problem or task Activity is undertaken with little or no prior formulation for a course of action Identifies short sequences of actions for a specific task Identifies potential multiple sequence routes for a complex task 19. Relationships (representing and formulating) Making connections between elements of knowledge Focused on (acts/shares) isolated pieces of information Building on input and information from others Integrates and synthesises of multiple pieces of information 18. Hypothesis “what if…” (reflecting and monitoring Changing from one line of reasoning or course of action to another as information or circumstances change Maintains a single line of approach tries multiple options in light of new information or lack of progress Reconstructs and reorganizes understanding of the problem in light of ne information or opinion. Task Regulation 15. Collecting ElementsExplore and understand Does not recognize the need for further information Identifies the need for specific information related o immediate activity Identifies need for varied information related to multiple activities. 13. Systematicity Implementing possible solutions to a monitoring progress Trial and error hypothesis testing in an unorganized sequence of solution attempts Forward search through a problem space with an organized sequence of solution attempts Forward and backward search through a problem space with reflective solution attempts 17. Tolerance for ambiguity Accepting ambiguous situations and exploring options within these Maintains only a presence in situations where there is ambiguity notes ambiguity and suggests options Explores the problem space 14. Organising (Problem analysis) Analysing and defining a problem in familiar language (i.e. Making the problem more manageable and meaningful) Problem is stated as initially represented either explicitly or implicitly Problem is divided into sub problems Problem is divided into sub problems and their inter dependence is recognised

36. Dimensionality

37. - cognitive

38. social

39. Developmental Progressions Social Strand Cognitive Strand Level 5- Cooperation, Shared Goals and Appreciation Level 5 Refined strategic application & problem solving Level 4 Mutual Commitment Level 4 Systematic and methodical Level 3 Resolving Differences Level 3 Strategic Planning Organising & Executing Level 2 Awareness of Partnership Level 2 directed approach Systematic Trial & Error Level 1 Clarification and Independence Level 1 Trial and Error Exploration

40. Feedback and Reporting

42. Reports

43. Reporting to Students Learning Readiness Real time reports will be available, based on empirically developed learning progressions. They will identify a point of readiness to learn for each student. This type of report will be linked to teaching interventions associated with readiness to learn as indicated by the black bar in the spine of the chart.

44. Instructional groups

45. Teacher development

46. Professional development modules 1.Defining and Assessing 21st Century Skills 2.Using a Developmental Model 3.ATC21S Assessments: Getting Started 4.Interpreting Reports 5.Teaching and Learning 21st Century Skills

47. The System An integrated system – Developmental progressions – standards referenced – Assessment tasks with automatic scoring – Teaching intervention and professional development – Reports for students, teachers and systems Next steps Access and Use Validation studies Expanded resources – mainstream curriculum Other 21 st century skills ATC21S Assessment and Teaching of 21st Century Skills The University of Melbourne Cisco Intel Microsoft www.atc21s.org

48. Transfer and Utility To what degree will 21st century skills such as collaborative problem solving or ICT Literacy in Learning Networks facilitate student learning? –Are these skills teachable and learnable? –Will the skills enhance learning in the classroom? To what degree might enhanced 21 st century skills contribute to adaptiveness of graduate students to the workforce?

49. ATC21s New Users?

50. Follow up!

51. Implications

52. Conventional Assessment Correctness if the important part of the response Response can indicate reasoning ability of a specific level Difficult to interpret various incorrect responses Additipnal data is lost in the focus on the correct response

53. Future Applications

54. Task design I.Design tasks to capture specific components of cognitive ability I.working memory II.Processing speed III.Pattern recognition IV.Systematicity’ II.Background data capture allows more user friendly and engaging asssessments I.Reduce test anxiety II.counteract teaching to the test III.Capture multiple variables IV.Reduce test time need V.Increase assessment efficiency VI.Better, richer information for teachers III.Applications in serious game industry

55. Roll Out and Dissemination

56. What is in the system? Two models - cloud and portable Prototype assessment tasks Developmental progressions Reporting and feedback for schools, teachers and students Professional development modules Specifications for developers

57. Portable and Cloud Systems Portable versionCloud version Local adaptationsCentralised control/management Decision-making autonomyCentralised decision-making Customisation enabled Embed or link within existing systems Local adaptations Addition of tasks No customisation Local support structureCentralised support structure Flexible registration systemRigid registration system Flash Less susceptible to differences across browsers Not currently supported on iPad* HTML 5 More susceptible to differences across browsers

58. PISA 2015 Collaborative problem solving to be tested in OECD countries in 2015. PISA will test human to computer interaction Collaborative Problem solving involving human to human maybe in 2018