1. 1 Final Presentation Spring 2006 T/TAC Online Project Virginia Assessments May 18, 2006 Matt Humphrey Aala’a Mashaal Pamela Sharpe Tim Smith Maria Washington Graduate School of Education George Mason University

2. 2 T/TAC Online Regions

3. 3 Our Faculty and Facilitators  Dr. Michael Behrmann  Dr. Nada Dabbagh  Dr. Kevin Clark  Anya Evmenova  Gary Dickelman  Jocelyn DelSignore

4. 4 Clients  John Eisenberg – Severe Disabilities/Assistive Technology Specialist, Virginia Department of Education  Mary Wilds – Statewide Coordinator of Distance Education, Old Dominion University  Dr. Michael Behrmann – Professor, Director of Helen A. Kellar Institute for Human disAbilities (KIHd), George Mason University  Nathan Sparks – Coordinator, Division of Assessment and Reporting, Virginia Department of Education

5. 5 Presentation Overview Part One Part Two Part Three Part Four Part Five Instructional Design Process Background & Introduction Mission and Goal of Project Analyses Design Development Formative Evaluation Prototype Demonstration Future T/TAC Site Recommendations Questions and Answers

6. 6 Project Background  No Child Left Behind Act (NCLB) 2002  Every child must be assessed according to the state standards  Individuals with Disabilities Education Improvement Act of 2004 (IDEA)  Accommodations for all children with disabilities

7. 7 Project Background  Assessment options for students who are exempt from traditional assessment programs  Virginia Alternate Assessment Program (VAAP)  Students assessed at grade levels K-5  Virginia Grade Level Assessment (VGLA)  Students assessed at grade levels 3-8  Virginia Substitute Evaluation Program (VSEP)  Students assessed at grade levels 9-12

8. 8 Project Problem Statement VA Assessments  Students with severe disabilities were only assessed on life skills prior to new mandates  Transition to cognitive standards based assessments  Communication skills are now considered in the assessment process  New process for teachers  Our project concentrates on the VAAP assessments

9. 9 Instructional Design Process The ADDIE Model AnalysisDesignDevelopmentImplementationEvaluation

10. 10 The Instructional Design Process  Who Learner  What Learner needs Instructional Strategies  How Evaluation

11. 11 AnalysisAnalysis Performance Analysis Needs Analysis Learner Analysis Task Analysis AnalysisDesignDevelopmentImplementationEvaluation

12. 12 Analysis - Findings Performance Analysis  New assessment process for teachers  Teachers were not using standardized procedures for assessments  Teachers did not have sufficient access to resources on the VA DOE website  Teachers were not targeting the Aligned Standards of Learning or communication skills standards

13. 13 Analysis - Findings Needs Analysis  Teachers must begin aligning lesson plans to new ASOL guidelines Include cognitive and communication skills Scaffolding needed  Skills will be measured in four subject areas: 1) Reading, 2) Math, 3) History & Social Science, 4) Science  Lesson plans will be scored on a scale of 1 to 4 A score of 3 is needed to qualify  Teachers should be able to differentiate and adapt existing SOL Enhanced lesson plans

14. 14 Analysis - Findings Learner Analysis  Varied experience levels amongst teachers to be addressed in solution system  Using the data collected from the surveys and interviews with SMEs, teachers, and administrators we found that our users were: Novice Special Education Teacher Intermediate Special Education Teacher Expert Special Education Teacher Reluctant Special Education Teacher T/TAC Online Administrator

15. 15 Analysis - Findings Task Analysis  Existing lesson plan document Existing lesson plan document  Instruction is focused on the process of creating lesson plans Support the teachers in aligning them to the new standards  Task analyze the lesson plan process Understand the process used by a special education teacher Support those processes in the solution system

16. 16 Analysis – Findings Task Analysis  Creating a lesson plan is a procedural task Problem solving tasks are embedded  Break down each step of the lesson plan creation process into its constituent parts  Identify what the system should provide to support teachers throughout the process Supports will be utilized in the solution system

17. 17 Instructional Objectives  Facilitate and support lesson plan creation process EPSS guides teachers through each step of creating a lesson plan Additional supports assist teachers throughout the process  Encourage knowledge sharing Lesson plan database  Allows teachers to search and download exemplary lesson plans Search VAAP Manual

18. 18 Design Design AnalysisDesignDevelopmentImplementationEvaluation Design Approach Storyboards

19. 19 Design Approach  Support the process of creating lesson plans aligned to new standards and allow knowledge sharing  Develop an electronic performance support system (EPSS) to be used by teachers  EPSS should include guides and necessary supports that will: Guide and scaffold the lesson plan creation process Allow special education teachers the ability to develop, share, and download lesson plans

20. 20 Design Approach Rationale  Supporting an existing process Teachers know how to create lesson plans New constraints on that process  Teacher experience levels vary  Just-in-time support in the classroom

21. 21 Design Approach EPSS Characteristics Electronic Performance Support System (EPSS)  Computer software program that improves performance by: Reducing complexity Providing the performance information to perform a task Providing a decision support system that enables the user to identify appropriate actions

22. 22 Design Approach EPSS Characteristics  Job aid  Controlled by the user, used on demand  Reduce the need for prior training in order to accomplish the task  Supports new constraints on an existing process

23. 23 Development AnalysisDesignDevelopmentImplementationEvaluation Flowcharts Prototype Usability Testing

24. 24 Development of Prototype  Operationalize instructional objectives  VAAP section will include three main sections: 1.Create Lesson Plans  Teachers will be guided through the lesson process with additional just-in-time supports 2.Search the VAAP Manual  Enable teachers to find VAAP information more efficiently 3.Search for Created Lesson Plans  Teachers can search and download exemplary lesson plans created by other teachers

25. 25 Flowchart 1: Searching the VAAP Manual

26. 26 Flowchart 2: Lesson Plan Document

27. 27 Flowchart 3: VA Assessments Site Flow

28. 28 Development Lesson Plan Generator (LPG)

29. 29 Lesson Plan Generator Final Prototype

30. 30 Evaluation Formative Evaluation The formative evaluation consisted of three stages of usability testing:  Stage one - novice users One on one, think aloud protocol  Stage two - experienced users. One on one, think aloud protocol  Stage three - expert users Small group, expert review

31. 31 Evaluation Results Positive Aspects of the System as Identified by the Testers:  Users could easily sign in with their existing accounts and new users could create an account easily  The navigation is clear and consistent with lesson plan objectives of the teachers simulating the real process for this task  The supports are appropriate for each section of the system with adequate information to support a teacher in creating a great lesson plan  The system is easy and clear in choosing organizing topics and having the standards auto populate dynamically is a necessary and strong feature

32. 32 Evaluation  System Improvements Usability Instructional Content Interface Design

33. 33 Evaluation Usability  Add a descriptor to the lesson plan generator system so users are aware of what they can do (This is an area where you can create your own lesson plan, ideas are provided through links on the right)  Make text throughout the site bolder with a sharper font for varying visual needs of users  Create a flashing button for “select” next to the organizing topic to assure that the users get the correct set of standard

34. 34 Development Changes Instructional Content  Create documents defining the standards and organizing topics  Reword the directions for the activities and strategies section to be clearer to the users  Add scoring rubrics to the evidence collection page to identify significance of peer interaction  Add examples and clarification documents to the evidence page  Remove the list of AT Vendors as it will not assist the user with the task at hand.

35. 35 Development Changes Interface Design  Made the Navigation bar have actual section titles as well as highlight where the user is in the system.  Made the logon screen bigger and the word “new user” larger to accommodate different visual needs  Made the user name and date auto populate based on logon user information.

36. 36 Next Steps - Recommendations General recommendations include the following: 1.Add a spell check feature. 2.Create a flashing button for “select” next to the organizing topic to assure that the users get the correct set of standard 3.Convert help files to PDF instead of Microsoft Word documents. 4.Add a Help button on Preview screen. Also an enlarge font button. 5.The age appropriate materials document should be changed to a pop-up box instead of a word document since the content is short to make loading it quicker. 6.Add visuals for better comprehension. 7.System implementation – experienced teachers entering lesson plans into the Lesson Plan Generator 8.Make the lesson plan search feature functional 9.Summative evaluation

37. 37 Acknowledgements  Shuangbao Wang – T/TAC Online Programmer  Seunghun Ok – T/TAC Online Site Administrator  MSHA Immersion Team  Kilmer Center - Vienna, VA  Courthouse Road Elementary – Spotsylvania County, VA  Vienna Elementary School – Vienna, VA  Assistive Technology Taskforce  T/TAC Staff

38. 38 Thank You for Listening Are there any questions?