1. Graduate School of Education George Mason University
Design Treatment Presentation Fall 2005 T/TAC Online Project VA Assessments December 15, 2005
Good afternoon. I am Pamela Sharpe one of the five members of the T/TAC Mission Six team. On behalf of the T/TAC team and the faculty of the GMU Immersion program we’d like to thank you for taking time to be here for the immersion students final presentation for this semester.
In September the members of my team were selected to work on the sixth grant awarded to GMU for the T/TAC Online project:
The members of the TTAC team include:
This design treatment is a summary of the progress we have made in the Graduate Instructional Technology Program on the T/TAC online project.
Matt Humphrey
Aala’a Mashaal
Pamela Sharpe
Tim Smith
Maria Washington
Graduate School of Education
George Mason University

2. Our Faculty and Facilitators
Dr. Nada Dabbagh
Dr. Kevin Clark
Dr. Mike Behrmann
Gary Dickelman
Jocelyn DelSignore
Anya Evmenova

3. Clients & Information Sources
Dr. Patricia Abrams
Dr. Michael Behrmann
John Eisenberg
Nathan M. Sparks
Mary Wilds
Human:
All clients
Jocelyn DelSignore
Anya Evmenova
Teachers
Content:
T/TAC & VA DOE websites
Assessment Manuals

4. Presentation Overview
Part One
Part Two
Part Three
Part Four
Part Five
Immersion Team Process
Instructional Design Process
Background & Introduction
Mission and Goal of Project
Analyses
Needs Analysis
Performance Analysis
Learners Analysis
Task Analysis
Findings and Results
Design Approach
Development
Next Steps
Questions and Answers

5. Immersion Team Process
The Learning Process
Teamwork
Portfolios and Reflections
Challenges
Benefits
As part of the learning process each team works for a client to produce an intervention using the IDP Process.
Teams are required to develop and manage a project website, which houses information about the team norms, vision and deliverables.
Individual team members are required to develop and maintain a personal website where class work, goals, and reflections are kept about the Immersion journey
Challenges:
Ill-structured, hands on
No formal assessments
Working for an actual client with project deliverables
Different personalities and skillsets
Additional assignments
Benefits:
Working in a team to solve problems and produce deliverables
Opportunity to gain skills in ID, Project Management and assume different roles
Real world environment
Coaching

6. The Instructional Design Process
Who
Learner
What
Learner needs
Instructional Strategies
How
Evaluation

7. Instructional Design Process Integrated Learning Design Framework

8. Instructional Design Process The ADDIE Model
Analysis
Develop
Implement
Evaluate

9. Project Background No Child Left Behind Act (NCLB)
Individuals with Disabilities Education Improvement Act of 2004 (IDEA)
Federal Mandates and VA Assessments
Standard of Learning (SOL) Assessment
Alternative SOL (VGLA and VSEP)
Alternate SOL (VAAP)
As a result of the NCLB act and the IDEA all students with disabilities must be considered for participation in the SOL Assessments.
No Child Left Behind Act is a federal legislation that requires states:
Demonstrate progress in raising the percentage of students who are proficient in reading & mathematics, and narrowing the achievement gap or risk a loss of government funding.
President Bused stated, “The Individuals with Disabilities Education Improvement Act of 2004 is designed to help children learn better.
Assessments are federal and state mandates. The state of Virginia uses three types of assessment tools….
Students assessed by the SOLs are in the general population
Alternative assessments - students who may need some modifications or accommodations in order to participate in standard statewide testing.
Alternate assessments are designed for students with more severe disabilities who cannot participate in regular assessment procedures even with modifications.

10. Training/Technical Assistance Centers (T/TAC)
T/TAC Online is a web-based community linking people and resources to help children and youth with disabilities
The Virginia Department of Education, is the sponsor of TTAC online and the TTAC. There are eight superintendent regions that are supported by TTAC.
TTACs are tasked with providing professional development assistance to special education teachers. TTACs are located throughout the state of Virginia and comprise a statewide system
T/TAC Online is available to teachers, parents and professionals who can utilize the services through the convenience of a personal computer. T/TAC Online provides a central location for the community to gather information and share knowledge.
Currently, the knowledge areas are categorized as SOL Enhanced, Resources, Events, Online Training, School Improvement, and VA Assessments. The site was designed and developed by George Mason University.
One of the goals of T/TAC Mission 6 Project involved adding an “ASSESSMENT” section to the T/TAC online website. This section is to provide a “one stop shop” on T/TAC online where teachers, administrators and parents can find information and resources on alternative and alternate assessments.
For this round of the T/TAC Online project we are focusing on the Virginia Alternative Assessment Program manual.

11. Project Problem Statement VA Assessments
Transition to Cognitive Standards Based Assessments
Communication Skills
State Mandated Assessments
New Process for Teachers

12. Analysis Analysis Needs Analysis Performance Analysis Learner Analysis
Design
Develop
Implement
Evaluate
Analysis
Needs Analysis
Performance Analysis
Learner Analysis
Task Analysis

13. Needs Analysis Overview
The process of determining, analyzing, and prioritizing needs and, in turn, identifying and implementing solution strategies to resolve high-priority needs.

14. Needs Analysis Purpose
Determine the best solution for T/TAC Online
Actuals and Optimals
Generate data to support in making a recommendation
Provide specific recommendations
Decide priorities for action

15. Needs Analysis Steps Taken
Determine Current Status
Performance Analysis
Analysis of Results
Teacher Expectations
Identify Discrepancies
Determine the Goals
Set Priorities for Action

16. Performance Analysis Overview
Used to determine what needs to be done in order to accomplish the client’s stated goals and objectives
Focuses on determining the elements that support the project
Investigates the current situation (actuals) and desired performance (optimals)

17. Performance Analysis Results
Initial Actuals
Teachers were not using standardized procedures for assessments
Teachers did not have sufficient access to resources on the VA DOE website
Teachers were not targeting the Aligned Standards of Learning

18. Performance Analysis Results
Initial Optimals
Teachers will use standardized procedures for assessment
Teachers will have easy access to resources
Teachers will target the ASOLs for assessment

19. Learner Analysis
A learner analysis is performed to determine who the users will be, what skills and knowledge are needed by the learners, the environment in which they work, and their motivations and goals for performing.

20. Learner Analysis Why Create Personas?
Personas identify the user motivations, expectations and goals responsible for driving behavior.
Although personas are fictitious, they are based on knowledge of real users. Some form of user research is conducted before they are written to ensure they represent end users rather than the opinion of the person writing the personas.
There are many varying personas that represent our users.

21. Learner Analysis How We Developed Our Personas
Using the data collected from the surveys and interviews with SMEs, teachers, and administrators we found that our users were:
Novice Special Education Teacher
Intermediate Special Education Teacher
Expert Special Education Teacher
Reluctant Special Education Teacher
T/TAC Online Administrator
Parent/Guardian

22. Needs Analysis Data Collection
Reviewed the VAAP Manual
Attend conferences to observe teacher’s reactions
Administer surveys
Visit schools to interact with special education teachers
Interview:
Dr. Behrmann
Mary Wilds
John Eisenberg
Jocelyn DelSignore
Anya Evmenova

23. Needs Analysis Survey
Gathered information in the following topic areas:
Demographics
Technology skills
Attitudes toward the VAAP
Attitudes toward new assessments

24. Needs Analysis Findings – New VAAP
Figure 1
As Tim stated our team created a survey instrument.
Given the constraints of time and resources, we had to be creative in determining how were going to administer it.
Currently there is a lot of activity around the new VAAP, and T/TAC staff are working diligently to provide training to teachers to help them make the transition.
As a result, of all the energy we were able to administer the instrument to a group of teachers attending a training on GMU’s campus and in Fredericksburg, VA.
However, we knew that the sample size and geographical distribution of the teachers surveyed would yield skewed results.
We decided that taking advantage of the opportunity to gather the much needed data on our learner population would be worth the risk.
Between the two training events we were able to collect 89 surveys and found that:
85.6% of the teachers indicated that they agree with the assessments.
67% of teachers surveyed indicated that they spent a lot of time assessing the students.
61% of teachers surveyed agreed that they have five or more tools for assessments
85.6% of the teachers indicated that they agree with the assessments.
67% of teachers surveyed indicated that they spent a lot of time assessing the students.
61% of teachers surveyed agreed that they have five or more tools for assessments
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

25. Needs Analysis Findings - Assessment
Figure 2
74% of teachers surveyed agreed that hindrances such as time, resources, and lack of appropriate information affect their assessment process. This statistic indicates that a solution system that would alleviates the hindrances would be welcomed
77% of teachers surveyed agreed that they would like to interact with other special education teachers also making the transition to the VAAP.
75% agreed that they would like to share lesson plans with other special education teachers
79%)agreed that they would like to get help aligning the ASOL and life skill components into their lesson plans
We provided space on the instrument for the teachers to write comments and found that while they welcome the structure that the VAAP will afford them, they find new process confusing.
74%
72%
77%
75%
79%
62%
50%
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

26. Needs Analysis Findings - Age Ranges
When we designed our survey instrument we decided to ask the teachers to provide their:
age,
number of years of teaching experience and
licensure status (indicating whether or not they have a provisional or professional license)
In the hopes that this information would give us an indication of how resistant they might be to:
changes in the procedures they followed,
their level of comfort with technology, and
the level of skill and knowledge they possess in teaching students with severe disabilities.
Therefore, we assumed that age might be an indicator of the level of comfort that the user would have with technology and changes in procedures or may have a need for mentoring and or coaching from older or more experienced teachers.
Our survey found that:
Approximately 24% (or 17 individuals) were 50 or more years old
25% were between the ages of 40 and 50
Another 24% were between 30 and 40
and 27% were between 20 and 30
However, we found that age had little if any correlation with comfort level with technology since half of the teachers surveyed use T/TAC online and 88% “have sufficient access to a computer.” Clearly age had little bearing on the level of comfort with technology that our subjects possessed.
This and the next two slides serve as illustrations of the few challenges we faced during this phase of the project. While minor, they did serve as a reminders of just how iterative the instructional design process is.
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

27. Needs Analysis Findings - Years of Teaching Experience
While extracting the data from the surveys we discovered that there were a few instances where a teacher might be between the ages of 40 and 50 but may
only have a a couple of years of experience teaching students that participate in VAAP.
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

28. Needs Analysis Findings - Licensure
Ultimately we found that age, numbers of year of experience and licensure status had little impact on a teachers ability to meet the performance requirements identified
in the next phase of the NEEDS ASSESSMENT.
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

29. Needs Analysis Current Situation/Actuals
Special Education teachers have not been doing assessments for their students in a standardized format
Special Education teachers’ collaboration outside the classroom is limited
The assessment documents are difficult to locate on the Virginia Department of Education website
The challenge of assessing special education students creates a barrier
Special education teachers of students with severe disabilities have not been required to conduct SOL based assessments
Teachers spend an inordinate amount of time developing portfolios which are not aligned with SOL expectations
During the NEEDS ANALYSIS phase we confirmed some of the initial findings we made during the PERFORMANCE ANALYSIS.
Specifically we confirmed that
Special Education teachers have not been doing assessments for their students in a standardized format
Special Education teachers collaboration outside the classroom is limited
The assessment documents are difficult to locate on the Virginia Department of Education website
Special education teachers of students with severe disabilities have not been required to conduct SOL based assessments
Teachers spend an inordinate amount of time developing portfolios which were not aligned with SOL expectations

30. Needs Analysis Desired Performance/Optimals
For students designated to be assessed under the VAAP, teachers must begin aligning lesson plans to new aligned standards of learning (ASOL) guidelines, which include cognitive skills.
Skills will be measured in the four subject areas: Reading, Math, History & Social Science and Science.
Teachers must develop lesson plans that align to SOL guidelines and will receive a rating of three 3 or above on a scale of 1-4.
Differentiate the SOL enhanced lesson plans that are currently available on the SOL Enhanced area of T/TAC Online for their students assessed under VAAP guidelines.
During the NEEDS ANALYSIS we explored the OPTIMALS in order to identify precisely what the teachers are expected to do and how.
As we’ve stated previously, teachers of students designated to be assessed under the VAAP must begin developing lesson plans that incorporate the aligned standards of learning (ASOL).
Student performance on the aligned standards of learning will be measured in four categories: Reading, Math, History & Social Science and Science.
Additionally, because students that are eligible to be assessed under the VAAP are unable to participate in standard pen and paper assessments evidence of student performance is collected on a continuous basis. Therefore lesson plans should elicit observable behaviors that demonstrate student performance as demonstrated through communication. There are 14 communications skills outlined in the VAAP manual.
Teachers must develop lesson plans that align to SOL guidelines and can be rated 3 (scale of 1-4) or above on the established scoring rubric in the areas of
student performance
2) communication context,
3) communication settings,
4) communication socialization, and
5) communication support.

31. Needs Analysis Desired Performance/Optimals
It is important to note that if a student does not perform well in the four subject areas they may still garner a passing score by performing well in the following communication areas:
Student performance
Context
Settings
Socialization
Support
It is important to note that if a student does not perform well in the four subject areas they may still garner a passing score by performing well in the following communication areas: student performance
context
settings
socialization
support

32. Needs Analysis Discrepancies in Teacher Performance
Teachers do not have experience in aligning their lesson plans to the new standards.
Some of the teachers do not have experience assessing using the former VAAP standards.
The greatest discrepancy lies in the issue that the cognitive skill assessment process is new to all teachers in Virginia.
Teachers have limited resources to help them with the assessment process.
Teachers have limited time outside the classroom to dedicate to this process.
Based upon the ACTUALS and OPTIMALS as they relate to teacher performance we were able to identify the discrepancies
Teachers do not have experience in aligning their lesson plans to the new standards.
Some of the teachers do not have experience assessing using the former VAAP standards.
The greatest discrepancy lies in the issue that this process is completely new to all teachers in the state of Virginia.
Teachers have limited resources to help them with the assessment process.
Teachers have limited time outside the classroom to dedicate to this process

33. Needs Analysis Desired Performance/Optimals
Use computer:
Download blank forms
Internet access
Be able to use word-processing software
Be familiar with state and district-wide assessment
Have knowledge of learner’s skill level, needs, and characteristics
Have knowledge/access to proven research based instruction techniques
Be able to develop lesson plans that fit the student’s current stage of learning for a particular skill.
In order to reach the optimal level of performance teachers will:
Use a computer to:
Download blank forms
Access the Internet
Use word-processing software
Be familiar with state and district-wide assessment
Have knowledge of learner’s skill level, needs, and characteristics
Have knowledge/access to proven- research based instruction techniques
Be able to develop lesson plans that fit the student’s current stage of learning for a particular skill.

34. Needs Analysis Desired Performance/Optimals
The solution system should:
Assist teachers with assessing their students using the different assessment procedures outlined in the VAAP, VGLA, and VSEP manuals
Allow teachers to access the documents in one location or a “one stop shop”
Facilitate the sharing of lesson plans with other teachers also involved in the process of creating lesson plans
Facilitate the submission of standardized assessment portfolios with streamlined evidences thereby reducing the amount of time teachers spend compiling evidence of student performance
The documents should be searchable for easy access and downloading
We not only identified the optimal performance of the teachers, we also identified the optimal performance of the system that we design should assist teachers with
assessing their students using the different assessment procedures outlined in the VAAP, VGLA, and VSEP manuals
The system should also allow teachers to access the documents in one location or a “one stop shop”
The documents should also be searchable for easy access and allow for easy downloading
Furthermore, the system should facilitate the sharing of lesson plans with other teachers to also involved in the process of creating and submitting lesson plans
And facilitate the submission of standardized assessment portfolios with streamlined evidences thereby reducing the amount of time teachers spend compiling evidence of student performance

35. Needs Analysis Priorities for Action
Analyze lesson plan process
Determine potential solution based on the needs
Create prototype of the solution system
Once the NEEDS ANALYSIS phase was completed, we were able to identify our PRIORITIES FOR ACTION which were:
Analyze lesson plan process
Based on the needs, determine potential solution
Create prototype of the solution system
These priorities transitioned us to the next phase of the project. I would now like to bring MATT HUMPHREY UP TO WALK YOU THROUGH THE STEPS WE TOOK during TASK ANALYSIS PHASE

36. Task Analysis Overview
The process of analyzing and articulating the kind of learning that you expect the learners to know how to perform
A blueprint of the instructional sequence and it assures the instructional designer that there are no gaps in the instruction

37. Task Analysis Lesson Plan Process
Instruction is focused on the process of creating lesson plans
Support the teachers in aligning them to the new standards
Task analyze the lesson plan process
Understand the process used by a special education teacher
Support those processes in the solution system

38. Task Analysis Information Processing Analysis (IPA)
Used most often for procedural and cognitive tasks
First step in decomposing or breaking down a goal into its constituent parts
Identifies what instruction is needed to attain the goal

39. Task Analysis IPA/Lesson Plan Process
Creating a lesson plan is a procedural task
Break down each step of the lesson plan creation process into its constituent parts
Identify what we need to provide to the teachers to support them throughout the process
Supports will be utilized in the solution system

40. Task Analysis What the Teacher Needs To Do
Complete prerequisite tasks
Review present level of performance, assessment data, and IEP team recommendations
Review student profile document
Identify communication skills
Identify the student’s entry points
Plot entry level points and communication skills

41. Task Analysis What the Teacher Needs To Do
Complete the lesson plan process
Identify a main theme or event of the lesson plan
Record the date of creation
Choose an ASOL subject matter
Organizing topic, standard used, support in activity context
Problem-solving cognitive task
4. Choose a communication skill that can be incorporated into the lesson
Can be used multiple times throughout the lesson

42. Task Analysis What the Teacher Needs To Do
5. Identify the student tasks and activities
Main tasks, sub tasks, prerequisites, instructional strategies
Rule-using cognitive task
6. Identify the materials needed
7. Identify the assistive technology needs
Ensure that all students have appropriate assistive technology throughout the lesson
8. Identify the evidence collections materials
Problem-solving cognitive task

43. Task Analysis Challenges
Is this the only lesson plan process?
Identifying types of cognitive tasks
Identifying appropriate supports for each task
Identifying instructional strategies
What does the final result of each task look like?

44. Task Analysis Lesson Plan Document

45. Solution System
Based upon the results of the needs and task analysis phases, our goals for the solution system are to:
Provide scaffolding for teachers to develop lesson plans aligned with the new SOLs.
Produce a searchable VAAP manual.
Provide additional resources for teachers to utilize throughout the assessment process.

46. Analysis  Design Design Analysis Develop Implement Evaluate
Performance Analysis
Needs Analysis
Personas
Task Analysis
Design Approach
Storyboards

47. Design Approach EPSS Characteristics
Electronic Performance Support System (EPSS)
Computer software program that improves performance by:
Reducing complexity
Providing the performance information to perform a task
Providing a decision support system that enables the user to identify appropriate actions

48. Design Approach EPSS Characteristics
Used on the job
Controlled by the user
Reduce the need for prior training in order to accomplish the task
Supports new constraints on an existing process

49. Design Approach Rationale
Supporting an existing process
Teachers have created lesson plans
New constraints on that process
New standard format for lesson plans is a system requirement
Teacher experience levels vary
Just-in-time support in the classroom

50. Design  Development Develop Analysis Design Implement Evaluate
Flowcharts
Schemas
Prototype

51. Development of Prototype
Users will be able to use the VA Assessments section on T/TAC Online to:
Search the VAAP Manual
Create Lesson Plans
Search for Created Lesson Plans

52. Searchable Online VAAP Manual
Users should be able to search the VAAP Manual by:
Subject
Subject and ASOL code
Subject and Skill Group
Instructional Strategy

53. Flowchart 1: Searching the VAAP Manual

55. Creating a Lesson Plan
Users will use the system to create lesson plans using a wizard with built in just-in-time supports
Users can also receive assistance if they need help with a particular part of the lesson plan creation process

56. Flowchart 2: Lesson Plan Document

57. Lesson Plan Database Schema

58. Searching for Created Lesson Plans
Users will be able to search for previously created lesson plans by:
Subject
Subject & Skill Group
Subject & ASOL Code

59. Design Approach Prototype

60. Next Steps Identify supports to be integrated into the EPSS
Interview expert teachers
Surveys
Resources
Collaboration
Continue Development of Prototype
Implement Prototype
Usability Testing
Formative Evaluation
Moving forward to the Spring of 2006 the TTAC team next steps will be to:
Working with expert teachers to identify additional supports and resources that would assist them in developing lesson plans that achieve a pass proficient score on assessments
This will include conducting additional site visits, distributing additional surveys and questionnaires and conducting interviews with teachers and SME’s.
Using these methods we hope to identify additional resources such as lesson plan documents, materials, assistive technology, and electronic tools which users would use to complete pass proficient lesson plans.
One of the stated client goals included improving collaboration among teachers, we will conduct further surveys and interviews with teachers, subject matter experts to determine what is the best type of web based collaboration for the users.
Continue developing the prototype by adding additional resources, lesson plans, materials and tools that assist users in reaching the goals.

61. Thank You for Listening
Are there any questions?