Graduate School of Education George Mason University Design Treatment Presentation Fall 2005 T/TAC Online Project VA Assessments December 15, 2005 Good afternoon. I am Pamela Sharpe one of the five members of the T/TAC Mission Six team. On behalf of the T/TAC team and the faculty of the GMU Immersion program we’d like to thank you for taking time to be here for the immersion students final presentation for this semester. In September the members of my team were selected to work on the sixth grant awarded to GMU for the T/TAC Online project: The members of the TTAC team include: This design treatment is a summary of the progress we have made in the Graduate Instructional Technology Program on the T/TAC online project. Matt Humphrey Aala’a Mashaal Pamela Sharpe Tim Smith Maria Washington Graduate School of Education George Mason University

Our Faculty and Facilitators Dr. Nada Dabbagh Dr. Kevin Clark Dr. Mike Behrmann Gary Dickelman Jocelyn DelSignore Anya Evmenova

Clients & Information Sources Dr. Patricia Abrams Dr. Michael Behrmann John Eisenberg Nathan M. Sparks Mary Wilds Human: All clients Jocelyn DelSignore Anya Evmenova Teachers Content: T/TAC & VA DOE websites Assessment Manuals

Presentation Overview Part One Part Two Part Three Part Four Part Five Immersion Team Process Instructional Design Process Background & Introduction Mission and Goal of Project Analyses Needs Analysis Performance Analysis Learners Analysis Task Analysis Findings and Results Design Approach Development Next Steps Questions and Answers

Immersion Team Process The Learning Process Teamwork Portfolios and Reflections Challenges Benefits As part of the learning process each team works for a client to produce an intervention using the IDP Process. Teams are required to develop and manage a project website, which houses information about the team norms, vision and deliverables. Individual team members are required to develop and maintain a personal website where class work, goals, and reflections are kept about the Immersion journey Challenges: Ill-structured, hands on No formal assessments Working for an actual client with project deliverables Different personalities and skillsets Additional assignments Benefits: Working in a team to solve problems and produce deliverables Opportunity to gain skills in ID, Project Management and assume different roles Real world environment Coaching

The Instructional Design Process Who Learner What Learner needs Instructional Strategies How Evaluation

Instructional Design Process Integrated Learning Design Framework

Instructional Design Process The ADDIE Model Analysis Develop Implement Evaluate

Project Background No Child Left Behind Act (NCLB) Individuals with Disabilities Education Improvement Act of 2004 (IDEA) Federal Mandates and VA Assessments Standard of Learning (SOL) Assessment Alternative SOL (VGLA and VSEP) Alternate SOL (VAAP) As a result of the NCLB act and the IDEA all students with disabilities must be considered for participation in the SOL Assessments. No Child Left Behind Act is a federal legislation that requires states: Demonstrate progress in raising the percentage of students who are proficient in reading & mathematics, and narrowing the achievement gap or risk a loss of government funding. President Bused stated, “The Individuals with Disabilities Education Improvement Act of 2004 is designed to help children learn better. Assessments are federal and state mandates. The state of Virginia uses three types of assessment tools…. Students assessed by the SOLs are in the general population Alternative assessments - students who may need some modifications or accommodations in order to participate in standard statewide testing. Alternate assessments are designed for students with more severe disabilities who cannot participate in regular assessment procedures even with modifications.

Training/Technical Assistance Centers (T/TAC) T/TAC Online is a web-based community linking people and resources to help children and youth with disabilities http://ttaconline.org The Virginia Department of Education, is the sponsor of TTAC online and the TTAC. There are eight superintendent regions that are supported by TTAC. TTACs are tasked with providing professional development assistance to special education teachers. TTACs are located throughout the state of Virginia and comprise a statewide system T/TAC Online is available to teachers, parents and professionals who can utilize the services through the convenience of a personal computer. T/TAC Online provides a central location for the community to gather information and share knowledge. Currently, the knowledge areas are categorized as SOL Enhanced, Resources, Events, Online Training, School Improvement, and VA Assessments. The site was designed and developed by George Mason University. One of the goals of T/TAC Mission 6 Project involved adding an “ASSESSMENT” section to the T/TAC online website. This section is to provide a “one stop shop” on T/TAC online where teachers, administrators and parents can find information and resources on alternative and alternate assessments. For this round of the T/TAC Online project we are focusing on the Virginia Alternative Assessment Program manual.

Project Problem Statement VA Assessments Transition to Cognitive Standards Based Assessments Communication Skills State Mandated Assessments New Process for Teachers

Analysis Analysis Needs Analysis Performance Analysis Learner Analysis Design Develop Implement Evaluate Analysis Needs Analysis Performance Analysis Learner Analysis Task Analysis

Needs Analysis Overview The process of determining, analyzing, and prioritizing needs and, in turn, identifying and implementing solution strategies to resolve high-priority needs.

Needs Analysis Purpose Determine the best solution for T/TAC Online Actuals and Optimals Generate data to support in making a recommendation Provide specific recommendations Decide priorities for action

Needs Analysis Steps Taken Determine Current Status Performance Analysis Analysis of Results Teacher Expectations Identify Discrepancies Determine the Goals Set Priorities for Action

Performance Analysis Overview Used to determine what needs to be done in order to accomplish the client’s stated goals and objectives Focuses on determining the elements that support the project Investigates the current situation (actuals) and desired performance (optimals)

Performance Analysis Results Initial Actuals Teachers were not using standardized procedures for assessments Teachers did not have sufficient access to resources on the VA DOE website Teachers were not targeting the Aligned Standards of Learning

Performance Analysis Results Initial Optimals Teachers will use standardized procedures for assessment Teachers will have easy access to resources Teachers will target the ASOLs for assessment

Learner Analysis A learner analysis is performed to determine who the users will be, what skills and knowledge are needed by the learners, the environment in which they work, and their motivations and goals for performing.

Learner Analysis Why Create Personas? Personas identify the user motivations, expectations and goals responsible for driving behavior. Although personas are fictitious, they are based on knowledge of real users. Some form of user research is conducted before they are written to ensure they represent end users rather than the opinion of the person writing the personas. There are many varying personas that represent our users.

Learner Analysis How We Developed Our Personas Using the data collected from the surveys and interviews with SMEs, teachers, and administrators we found that our users were: Novice Special Education Teacher Intermediate Special Education Teacher Expert Special Education Teacher Reluctant Special Education Teacher T/TAC Online Administrator Parent/Guardian

Needs Analysis Data Collection Reviewed the VAAP Manual Attend conferences to observe teacher’s reactions Administer surveys Visit schools to interact with special education teachers Interview: Dr. Behrmann Mary Wilds John Eisenberg Jocelyn DelSignore Anya Evmenova

Needs Analysis Survey Gathered information in the following topic areas: Demographics Technology skills Attitudes toward the VAAP Attitudes toward new assessments

Needs Analysis Findings – New VAAP Figure 1 As Tim stated our team created a survey instrument. Given the constraints of time and resources, we had to be creative in determining how were going to administer it. Currently there is a lot of activity around the new VAAP, and T/TAC staff are working diligently to provide training to teachers to help them make the transition. As a result, of all the energy we were able to administer the instrument to a group of teachers attending a training on GMU’s campus and in Fredericksburg, VA. However, we knew that the sample size and geographical distribution of the teachers surveyed would yield skewed results. We decided that taking advantage of the opportunity to gather the much needed data on our learner population would be worth the risk. Between the two training events we were able to collect 89 surveys and found that: 85.6% of the teachers indicated that they agree with the assessments. 67% of teachers surveyed indicated that they spent a lot of time assessing the students. 61% of teachers surveyed agreed that they have five or more tools for assessments 85.6% of the teachers indicated that they agree with the assessments. 67% of teachers surveyed indicated that they spent a lot of time assessing the students. 61% of teachers surveyed agreed that they have five or more tools for assessments Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

Needs Analysis Findings - Assessment Figure 2 74% of teachers surveyed agreed that hindrances such as time, resources, and lack of appropriate information affect their assessment process. This statistic indicates that a solution system that would alleviates the hindrances would be welcomed 77% of teachers surveyed agreed that they would like to interact with other special education teachers also making the transition to the VAAP. 75% agreed that they would like to share lesson plans with other special education teachers 79%)agreed that they would like to get help aligning the ASOL and life skill components into their lesson plans We provided space on the instrument for the teachers to write comments and found that while they welcome the structure that the VAAP will afford them, they find new process confusing. 74% 72% 77% 75% 79% 62% 50% Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

Needs Analysis Findings - Age Ranges When we designed our survey instrument we decided to ask the teachers to provide their: age, number of years of teaching experience and licensure status (indicating whether or not they have a provisional or professional license) In the hopes that this information would give us an indication of how resistant they might be to: changes in the procedures they followed, their level of comfort with technology, and the level of skill and knowledge they possess in teaching students with severe disabilities. Therefore, we assumed that age might be an indicator of the level of comfort that the user would have with technology and changes in procedures or may have a need for mentoring and or coaching from older or more experienced teachers. Our survey found that: Approximately 24% (or 17 individuals) were 50 or more years old 25% were between the ages of 40 and 50 Another 24% were between 30 and 40 and 27% were between 20 and 30 However, we found that age had little if any correlation with comfort level with technology since half of the teachers surveyed use T/TAC online and 88% “have sufficient access to a computer.” Clearly age had little bearing on the level of comfort with technology that our subjects possessed. This and the next two slides serve as illustrations of the few challenges we faced during this phase of the project. While minor, they did serve as a reminders of just how iterative the instructional design process is. Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

Needs Analysis Findings - Years of Teaching Experience While extracting the data from the surveys we discovered that there were a few instances where a teacher might be between the ages of 40 and 50 but may only have a a couple of years of experience teaching students that participate in VAAP. Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

Needs Analysis Findings - Licensure Ultimately we found that age, numbers of year of experience and licensure status had little impact on a teachers ability to meet the performance requirements identified in the next phase of the NEEDS ASSESSMENT. Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

Needs Analysis Current Situation/Actuals Special Education teachers have not been doing assessments for their students in a standardized format Special Education teachers’ collaboration outside the classroom is limited The assessment documents are difficult to locate on the Virginia Department of Education website The challenge of assessing special education students creates a barrier Special education teachers of students with severe disabilities have not been required to conduct SOL based assessments Teachers spend an inordinate amount of time developing portfolios which are not aligned with SOL expectations During the NEEDS ANALYSIS phase we confirmed some of the initial findings we made during the PERFORMANCE ANALYSIS. Specifically we confirmed that Special Education teachers have not been doing assessments for their students in a standardized format Special Education teachers collaboration outside the classroom is limited The assessment documents are difficult to locate on the Virginia Department of Education website Special education teachers of students with severe disabilities have not been required to conduct SOL based assessments Teachers spend an inordinate amount of time developing portfolios which were not aligned with SOL expectations

Needs Analysis Desired Performance/Optimals For students designated to be assessed under the VAAP, teachers must begin aligning lesson plans to new aligned standards of learning (ASOL) guidelines, which include cognitive skills. Skills will be measured in the four subject areas: Reading, Math, History & Social Science and Science. Teachers must develop lesson plans that align to SOL guidelines and will receive a rating of three 3 or above on a scale of 1-4. Differentiate the SOL enhanced lesson plans that are currently available on the SOL Enhanced area of T/TAC Online for their students assessed under VAAP guidelines. During the NEEDS ANALYSIS we explored the OPTIMALS in order to identify precisely what the teachers are expected to do and how. As we’ve stated previously, teachers of students designated to be assessed under the VAAP must begin developing lesson plans that incorporate the aligned standards of learning (ASOL). Student performance on the aligned standards of learning will be measured in four categories: Reading, Math, History & Social Science and Science. Additionally, because students that are eligible to be assessed under the VAAP are unable to participate in standard pen and paper assessments evidence of student performance is collected on a continuous basis. Therefore lesson plans should elicit observable behaviors that demonstrate student performance as demonstrated through communication. There are 14 communications skills outlined in the VAAP manual. Teachers must develop lesson plans that align to SOL guidelines and can be rated 3 (scale of 1-4) or above on the established scoring rubric in the areas of student performance 2) communication context, 3) communication settings, 4) communication socialization, and 5) communication support.

Needs Analysis Desired Performance/Optimals It is important to note that if a student does not perform well in the four subject areas they may still garner a passing score by performing well in the following communication areas: Student performance Context Settings Socialization Support It is important to note that if a student does not perform well in the four subject areas they may still garner a passing score by performing well in the following communication areas: student performance context settings socialization support

Needs Analysis Discrepancies in Teacher Performance Teachers do not have experience in aligning their lesson plans to the new standards. Some of the teachers do not have experience assessing using the former VAAP standards. The greatest discrepancy lies in the issue that the cognitive skill assessment process is new to all teachers in Virginia. Teachers have limited resources to help them with the assessment process. Teachers have limited time outside the classroom to dedicate to this process. Based upon the ACTUALS and OPTIMALS as they relate to teacher performance we were able to identify the discrepancies Teachers do not have experience in aligning their lesson plans to the new standards. Some of the teachers do not have experience assessing using the former VAAP standards. The greatest discrepancy lies in the issue that this process is completely new to all teachers in the state of Virginia. Teachers have limited resources to help them with the assessment process. Teachers have limited time outside the classroom to dedicate to this process

Needs Analysis Desired Performance/Optimals Use computer: Download blank forms Internet access Be able to use word-processing software Be familiar with state and district-wide assessment Have knowledge of learner’s skill level, needs, and characteristics Have knowledge/access to proven research based instruction techniques Be able to develop lesson plans that fit the student’s current stage of learning for a particular skill. In order to reach the optimal level of performance teachers will: Use a computer to: Download blank forms Access the Internet Use word-processing software Be familiar with state and district-wide assessment Have knowledge of learner’s skill level, needs, and characteristics Have knowledge/access to proven- research based instruction techniques Be able to develop lesson plans that fit the student’s current stage of learning for a particular skill.

Needs Analysis Desired Performance/Optimals The solution system should: Assist teachers with assessing their students using the different assessment procedures outlined in the VAAP, VGLA, and VSEP manuals Allow teachers to access the documents in one location or a “one stop shop” Facilitate the sharing of lesson plans with other teachers also involved in the process of creating lesson plans Facilitate the submission of standardized assessment portfolios with streamlined evidences thereby reducing the amount of time teachers spend compiling evidence of student performance The documents should be searchable for easy access and downloading We not only identified the optimal performance of the teachers, we also identified the optimal performance of the system that we design should assist teachers with assessing their students using the different assessment procedures outlined in the VAAP, VGLA, and VSEP manuals The system should also allow teachers to access the documents in one location or a “one stop shop” The documents should also be searchable for easy access and allow for easy downloading Furthermore, the system should facilitate the sharing of lesson plans with other teachers to also involved in the process of creating and submitting lesson plans And facilitate the submission of standardized assessment portfolios with streamlined evidences thereby reducing the amount of time teachers spend compiling evidence of student performance

Needs Analysis Priorities for Action Analyze lesson plan process Determine potential solution based on the needs Create prototype of the solution system Once the NEEDS ANALYSIS phase was completed, we were able to identify our PRIORITIES FOR ACTION which were: Analyze lesson plan process Based on the needs, determine potential solution Create prototype of the solution system These priorities transitioned us to the next phase of the project. I would now like to bring MATT HUMPHREY UP TO WALK YOU THROUGH THE STEPS WE TOOK during TASK ANALYSIS PHASE

Task Analysis Overview The process of analyzing and articulating the kind of learning that you expect the learners to know how to perform A blueprint of the instructional sequence and it assures the instructional designer that there are no gaps in the instruction

Task Analysis Lesson Plan Process Instruction is focused on the process of creating lesson plans Support the teachers in aligning them to the new standards Task analyze the lesson plan process Understand the process used by a special education teacher Support those processes in the solution system

Task Analysis Information Processing Analysis (IPA) Used most often for procedural and cognitive tasks First step in decomposing or breaking down a goal into its constituent parts Identifies what instruction is needed to attain the goal

Task Analysis IPA/Lesson Plan Process Creating a lesson plan is a procedural task Break down each step of the lesson plan creation process into its constituent parts Identify what we need to provide to the teachers to support them throughout the process Supports will be utilized in the solution system

Task Analysis What the Teacher Needs To Do Complete prerequisite tasks Review present level of performance, assessment data, and IEP team recommendations Review student profile document Identify communication skills Identify the student’s entry points Plot entry level points and communication skills

Task Analysis What the Teacher Needs To Do Complete the lesson plan process Identify a main theme or event of the lesson plan Record the date of creation Choose an ASOL subject matter Organizing topic, standard used, support in activity context Problem-solving cognitive task 4. Choose a communication skill that can be incorporated into the lesson Can be used multiple times throughout the lesson

Task Analysis What the Teacher Needs To Do 5. Identify the student tasks and activities Main tasks, sub tasks, prerequisites, instructional strategies Rule-using cognitive task 6. Identify the materials needed 7. Identify the assistive technology needs Ensure that all students have appropriate assistive technology throughout the lesson 8. Identify the evidence collections materials Problem-solving cognitive task

Task Analysis Challenges Is this the only lesson plan process? Identifying types of cognitive tasks Identifying appropriate supports for each task Identifying instructional strategies What does the final result of each task look like?

Task Analysis Lesson Plan Document

Solution System Based upon the results of the needs and task analysis phases, our goals for the solution system are to: Provide scaffolding for teachers to develop lesson plans aligned with the new SOLs. Produce a searchable VAAP manual. Provide additional resources for teachers to utilize throughout the assessment process.

Analysis  Design Design Analysis Develop Implement Evaluate Performance Analysis Needs Analysis Personas Task Analysis Design Approach Storyboards

Design Approach EPSS Characteristics Electronic Performance Support System (EPSS) Computer software program that improves performance by: Reducing complexity Providing the performance information to perform a task Providing a decision support system that enables the user to identify appropriate actions

Design Approach EPSS Characteristics Used on the job Controlled by the user Reduce the need for prior training in order to accomplish the task Supports new constraints on an existing process

Design Approach Rationale Supporting an existing process Teachers have created lesson plans New constraints on that process New standard format for lesson plans is a system requirement Teacher experience levels vary Just-in-time support in the classroom

Design  Development Develop Analysis Design Implement Evaluate Flowcharts Schemas Prototype

Development of Prototype Users will be able to use the VA Assessments section on T/TAC Online to: Search the VAAP Manual Create Lesson Plans Search for Created Lesson Plans

Searchable Online VAAP Manual Users should be able to search the VAAP Manual by: Subject Subject and ASOL code Subject and Skill Group Instructional Strategy

Flowchart 1: Searching the VAAP Manual

Creating a Lesson Plan Users will use the system to create lesson plans using a wizard with built in just-in-time supports Users can also receive assistance if they need help with a particular part of the lesson plan creation process

Flowchart 2: Lesson Plan Document

Lesson Plan Database Schema

Searching for Created Lesson Plans Users will be able to search for previously created lesson plans by: Subject Subject & Skill Group Subject & ASOL Code

Design Approach Prototype http://immersion.gmu.edu/ttac/fall2005/epss

Next Steps Identify supports to be integrated into the EPSS Interview expert teachers Surveys Resources Collaboration Continue Development of Prototype Implement Prototype Usability Testing Formative Evaluation Moving forward to the Spring of 2006 the TTAC team next steps will be to: Working with expert teachers to identify additional supports and resources that would assist them in developing lesson plans that achieve a pass proficient score on assessments This will include conducting additional site visits, distributing additional surveys and questionnaires and conducting interviews with teachers and SME’s. Using these methods we hope to identify additional resources such as lesson plan documents, materials, assistive technology, and electronic tools which users would use to complete pass proficient lesson plans. One of the stated client goals included improving collaboration among teachers, we will conduct further surveys and interviews with teachers, subject matter experts to determine what is the best type of web based collaboration for the users. Continue developing the prototype by adding additional resources, lesson plans, materials and tools that assist users in reaching the goals.

Thank You for Listening Are there any questions?