1. 2015 MCAS-Alt: How to Submit a “Complete” Portfolio
Karen Orlando
MCAS-Alt Teacher Consultant

2. Topics Top five reasons portfolios were scored Incomplete
Other reasons for an Incomplete score
Checking for “completeness”
Frequently-asked questions
Massachusetts Department of Elementary and Secondary Education

3. Top Five Reasons for Incomplete Portfolios
Massachusetts Department of Elementary and Secondary Education

4. Top Five Reasons For Incomplete Portfolios
Some or all brief description(s) did not address the measurable outcome. As a result, student did not address measurable outcome on at least 8 dates.
Brief descriptions were not included or did not provide enough information on at least 8 dates.
Titles and/or samples of text used with student were missing in the ELAReading strand.
At least two pieces of primary evidence were not submitted that related to measurable outcome listed on the data chart.
No evidence of self-evaluation in the strand.
Massachusetts Department of Elementary and Secondary Education

5. Top Five Reasons For Incomplete Portfolios
Some or all brief description(s) did not address the measurable outcome. As a result, student did not address the measurable outcome on at least 8 dates.
Massachusetts Department of Elementary and Secondary Education

6. Walter will identify basic forms of energy with 75% accuracy and 75% independence.
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

7. Student will answer simple comprehension questions to demonstrate the understanding of an informational text with 80% accuracy and 80% independence.
Print FULL SCREEN
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

8. Brief Descriptions Unmatched to Measurable Outcome
Page 35
2015 Educator’s Manual for MCAS-Alt
Brief Descriptions Unmatched to Measurable Outcome
Page 35: “…include a brief description beneath each data point that clearly describes what the student did and how the student addressed the skill, taking particular care to document that the activity matched the measurable outcome .”
Page 35: “…assess and record data only on those skills directly based on the measurable outcome. The data will be inconclusive, and the chances of scoring Incomplete will increase, when unrelated or multiple skills are included on the same data chart.
Massachusetts Department of Elementary and Secondary Education

9. Creating a Measurable Outcome from Entry Points that Include Related Skills
Example:
Entry Point as written:
“Solve number sentences that represent one-step multiplication and division word problems.”
Option 1:
Create a measurable outcome that addresses both skills.
All activities must document “solving number sentences involving one-step multiplication AND division problems.”
Option 2:
Modify the measurable outcome: “Solve number sentences that represent one-step multiplication word problems.”
All activities must document “solving number sentences involving one-step multiplication problems.”
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

10. Which descriptions are scorable?
Measurable Outcome: Student will solve addition and subtraction word problems with 80% accuracy and 100% independence.
Student solved addition and subtraction word problems worksheet.
Student solved subtraction word problems in math class.
Word problems game
with whole class
Solved addition and subtraction word problems created by peers on smart board
Solved addition problems on computer with no help
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

11. Clear and Specific Descriptions
Score accuracy and independence only on the skill related to the measurable outcome. Clearly indicate this on the work description label.
Example:
Skill measured in the measurable outcome:
“Answer questions about bar graphs”
Activity: Student creates a bar graph from the results of a class survey, then answers questions about the graph.
Scorable Description: “After creating a bar graph from a class survey, student answered questions about the graph. (Accuracy and independence were based on questions only.)”
Unscorable Description: “Student created a bar graph and answered questions.”
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

12. Top Five Reasons For Incomplete Portfolios
2. Brief descriptions were not included or did not provide enough information on at least 8 dates.
Massachusetts Department of Elementary and Secondary Education

13. Walter will compare fractions using the terms “greater than,” “less than, or “equal to” with 80% accuracy and 80% independence.
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

14. Scorable Brief Descriptions
Pages 35 and 40
2015 Educator’s Manual for MCAS-Alt
Scorable Brief Descriptions
Page 35: “include a brief description beneath each data point that clearly describes what the student did and how the student addressed the skill, taking particular care to document that the activity matched the measurable outcome;
Massachusetts Department of Elementary and Secondary Education

15. Scorable Brief Descriptions continued
Pages 35 and 40
2015 Educator’s Manual for MCAS-Alt
Scorable Brief Descriptions continued
Page 40: “Brief descriptions are provided below each learning activity that match the measurable outcome, and describe what the student was asked to do to address the measurable outcome, and how they did it. The following are examples of appropriate brief descriptions:…
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

16. Acceptable Brief Descriptions
Measurable Outcome for STE- Technology/Engineering
Walter will identify simple machines with 80% accuracy and 100% independence.
Date (m/d/y)
9/13/12
10/15/12
10/17/12
10/24/12
11/4/12
11/15/12
11/20/12
12/7/12
What the student did (skill)
How did they do it? (approach materials)
Identified simple machines by labeling pictures in an adapted textbook
After listening to “Simple Machines,”
identified
simple machines
in the book by pointing
Homework:
Identified simple machines at home, made a list of the ones he found
Worksheet
identified inclined planes and levers as simple machines using pictures and text
After watching video, Wheels and Axles,
identified wheels and axles around the school
Using Home Depot flyer, Walter identified levers and wedges using a bingo dauber
After watching Eduhead on the computer, Walter identified inclined planes by matching them to the pictures in the video
Class created a poster of simple machines he identified the simple machines in of each picture with a label
Massachusetts Department of Elementary and Secondary Education

17. Solve word problems on worksheet
These brief descriptions DO NOT provide enough information to know what the student was asked to do, or how he/she did it.
Brief
Description
Pre-test
Solve word problems on worksheet
Home-work
Digestive system class assign-
ment
Answer vocab questions
Label parts of digestive system
Describe digestive system
Chap. 2 test
Massachusetts Department of Elementary and Secondary Education

18. Which descriptions are scorable?
Outcome: After listening to story or article, student will answer simple comprehension questions to demonstrate understanding of an informational text with 80% accuracy and 80% independence.
Student answered questions about a story during speech class
Student answered questions on an informational text
After listening to Boston Globe article on “Pollution,” student answered simple comp. ques.
After listening to chapter 1 of Helen Keller by M. Davidson, student answered simple comp. ques.
Comprehension question quiz, chapter 2
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

19. Top Five Reasons For Incomplete Portfolios
Titles and/or samples of text used with student were missing in the ELAReading strand
Massachusetts Department of Elementary and Secondary Education

20. Titles of text are missing!
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

21. Cite the Title or Include Actual Text
Pages and 41
2015 Educator’s Manual for MCAS-Alt
Cite the Title or Include Actual Text
Page 15-19: “The title or a photocopy of the text used for assessment must be included in all brief descriptions for ELAReading; a copy of any teacher-created text must be submitted”
Page 41: “Evidence or brief descriptions in the ELAReading strand must refer by name to the text from which words, phrases, or excerpts were selected for assessment, and must include either:
the title of the published text, or
the actual text (e.g., sentence, passage, narrative, etc.), or
a photocopy or printout, if text is teacher-created, untitled, or includes selections from a print or digital source (e.g., websites)
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

22. Options for Citing the Text
Provide titles.
Provide copy of teacher-created text, worksheets, or selections from websites.
Attach a list of text used to data chart with dates and names of text used.
Include topic of the text if it is from a specific reading program, rather than just name of program.
Include title of text and/or topic if it is from a specific website, rather than just the name of the website.
For untitled texts include a photocopy
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

23. Top Five Reasons For Incomplete Portfolios
At least two pieces of primary evidence were not submitted that relate to the measurable outcome listed on the Strand Cover Sheet
Massachusetts Department of Elementary and Secondary Education

24. Student will compare and contrast characters in a story with 90%accuracy and 80%Independence
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

25. EVIDENCE #1
Student
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

26. Student is answering questions about characters,
EVIDENCE #2
Student is answering questions about characters,
not comparing and contrasting them.
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

27. Two Pieces of Primary Evidence
Pages and 33
2015 Educator’s Manual for MCAS-Alt
Two Pieces of Primary Evidence
Pages “…two additional pieces of primary evidence showing the student’s performance of the same targeted skill identified on the data chart.”
Page 33 “A minimum of one data chart and two pieces of additional primary evidence (called the “core set of evidence”)…”
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

28. Top Five Reasons For Incomplete Portfolios
5. No evidence of self-evaluation was found in this strand.
Massachusetts Department of Elementary and Secondary Education

29. Self-Evaluation Either… Evidence was either missing altogether, or
Showed no evidence of student choice
Stickers
Stamp
“Student smiled”
“Student traded in for music”
Massachusetts Department of Elementary and Secondary Education

30. Pages 39
2015 Educator’s Manual for MCAS-Alt
Self-Evaluation
“Self-evaluation activities document the student’s choices, decisions, and involvement before, during, and after instruction, including evidence that the student performed…”
Massachusetts Department of Elementary and Secondary Education

31. Examples of Self-Evaluation include:
planning and goal setting
choosing an activity or next steps in an activity
selecting a problem-solving strategy
monitoring own progress or use of a strategy (e.g., by checking off steps as each is completed)
deciding when to continue or end participation in an activity
identifying and correcting his/her own errors
graphing own performance or progress on a chart
determining own score using a rubric
selecting work for his/her own portfolio
reflecting on his or her performance; for example, teacher asks student to respond to:
What did I do well? What am I good at? Was this too easy?
What did we do during this activity? What did I learn?
How could I do better? Where do I need help?
What should I work on next? What would I like to learn?
Massachusetts Department of Elementary and Secondary Education

32. Massachusetts Department of Elementary and Secondary Education

33. Other Reasons for Incomplete
Literature and Informational text were combined in a single strand. The Reading strand must be one or the other.
Data chart did not document student performance on at least 8 different dates.
Both Accuracy and Independence began above %.
No data chart was submitted.
Data were unclear or could not be interpreted by the scorer.
Percent accuracy and/or independence were not included on primary evidence and/or could not be determined by scorer.
Student's name and/or date (month, day, year) was missing on one or more pieces of primary evidence, or was incorrect on Work Sample Description label.
Massachusetts Department of Elementary and Secondary Education

34. Level of Complexity = 1
Massachusetts Department of Elementary and Secondary Education

35. What Does Level of Complexity = 1 Mean?
Page 48
Educator’s Manual for MCAS-Alt
What Does Level of Complexity = 1 Mean?
Portfolio strand reflects little or no basis in,
or is unmatched to, Curriculum Framework learning standards required for assessment.
Strand must assess standards required for a student in that grade.
Data chart must document student performance of an acceptable measurable outcome in the required strand/domain. (Teachers cannot make up or use outdated entry points!)
ELA Reading is comprehension of words, phrases, sentences, passages based on a text.
Access skill must be taught in the context of an academic activity.
Massachusetts Department of Elementary and Secondary Education

36. Skill Not Taught in the Context of an Academic Activity
Massachusetts Department of Elementary and Secondary Education

37. Earth and Space Science is not assessed in High School
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

38. Checking for Completeness
Massachusetts Department of Elementary and Secondary Education

39. 12 Key Questions to Check for Completeness
Pages 35-36,
Educator’s Manual for MCAS-Alt
12 Key Questions to Check for Completeness
Does the strand include a data chart and at least two pieces of primary evidence
Is the first data point on the data chart below 80 percent accuracy and/or independence?
Does the data chart have at least 8 different valid dates?
Is the student’s name, date, % accuracy, and % independence listed on the data chart and on at least two pieces of primary evidence (or on a Work Description label)?
Massachusetts Department of Elementary and Secondary Education

40. 12 Key Questions to Check for Completeness (continued)
5. Do brief descriptions on the data chart state what the student was asked to do (i.e., what was the activity?) and how the activity was conducted (instructional approach, materials, etc.) on at least 8 different dates ? 6. Do brief descriptions on at least 8 different dates address the skill identified in the measurable outcome? 7. Do at least two pieces of primary evidence directly address the measurable outcome?
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

41. 12 Key Questions to Check for Completeness (continued)
8. Are all skills listed in the measurable outcome (e.g., “addition and subtraction”) addressed in at least 8 activities on the data chart and two pieces of evidence (all skills must be addressed in each activity). 9. If photographs or video are included as primary evidence, do they show a final product and are they clearly labeled? 10. If a teacher-scribed work sample is included as primary evidence, does it provide sufficient information to determine what the student did and how the student did it?
Massachusetts Department of Elementary and Secondary Education

42. 12 Key Questions to Check for Completeness
For ELA-Reading Strands Only:
11. Do brief descriptions include titles of all texts? 12. Do all activities in the strand include either Informational or Literary text (but not both)?
Massachusetts Department of Elementary and Secondary Education

43. Common Questions
Massachusetts Department of Elementary and Secondary Education

44. How Do I Determine Accuracy and Independence?
First, determine what the outcome is.
What are you asking the student to do?
Then, determine the activity the student will perform to address that outcome.
Next, divide that activity into “items” or steps.
Determine the overall accuracy (correctness) and independence (i.e., no prompts needed) for the entire activity:
% Independence = Number of “items” performed independently
Number of possible responses
Massachusetts Department of Elementary and Secondary Education

45. Page 38
2015 Educator’s Manual for MCAS-Alt
Examples of “Items”: Outcome: Andre will answer simple comprehension questions in relation to a text with 80% accuracy and 100% independence. Activity: After reading Chapter 1 in “Frog and Toad Are Friends” by A. Lobel, with support of visuals and a reading checklist, student answered 4 comprehension questions on a worksheet.
Question#1
Correct (accurate)
Visual prompt
(not independent)
Question#2
Correct(accurate)
No Prompt (independent)
Question#3
Gestural prompt
Question#4
Incorrect (inaccurate)
No prompt
(independent)
Averages for all activities
3/4 or 75% accurate
2/4 or 50% independent
Massachusetts Department of Elementary and Secondary Education

46. Massachusetts Department of Elementary and Secondary Education

47. How many correct responses? 2
How many questions? 5
How many correct responses? 2 P
What is the accuracy score? 2/5 or 40% I I
How many independent responses? 4 I
What is the independence score? 4/5 or 80% I
Massachusetts Department of Elementary and Secondary Education

48. Independence: Prompts vs. Accommodations
Pages 37-39, 50
Educator’s Manual for MCAS-Alt
Independence: Prompts vs. Accommodations
The following are cues and prompts since they guide the student toward the answer:
Verbal assistance (coach or tell: ”look on page…”)
Gesture (point or indicate where to find answer)
Visual cue (provide a sample problem or model)
Full or partial physical assistance (including hand-over-hand assistance)
Any Cue or Prompt = a “non-independent response”
Massachusetts Department of Elementary and Secondary Education

49. What is Generalized Performance?
Page 39,
2015 Educator’s Manual for MCAS-Alt
What is Generalized Performance?
2015 Educator’s Manual:
Page 39: “Generalized performance reflects the student’s ability to transfer knowledge and/or skills to other learning situations through application of a range of instructional approaches and activity formats, including any of the following instructional elements:
media and materials (e.g., using art materials, written text, manipulatives, computer)
activity formats (e.g., classroom projects, small group discussions, paired research, experiments)
presentation formats (e.g., oral, written, multimedia)
method of response (e.g., handwritten, word-processed, oral, creation of a visual display, video)”
Massachusetts Department of Elementary and Secondary Education

50. What is a Teacher-Scribed Work Sample and When Should it Be Used?
Page 37,
2015 Educator’s Manual for MCAS-Alt
What is a Teacher-Scribed Work Sample and When Should it Be Used?
One method of documenting performance of a student who does not produce written work
Indicates what the activity was, what response options were offered to student, and how he/she responded on each item (accurately, independently?)
Documents student’s performance of a series of tasks recorded by the teacher at one time.
Massachusetts Department of Elementary and Secondary Education

51. Teacher-Scribed Work Sample
Activity
What was the activity?
What materials were used?
What was the correct response?
What was the student’s actual response?
Student’s Response
Materials
Correct Response
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

52. Teacher-Scribed Work Sample: Not Enough Information to Score
What was the activity?
What was the correct response?
What was the student’s actual response?
What materials were used?
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

53. Reminders Double-check the evidence before submitting
Have another teacher or supervisor double-check your evidence and see if they can answer “yes” to the 12 questions.
Check for clerical errors such as incorrect dates, names or work of another student, or evidence submitted in wrong strand.
Attend Portfolios-in-Progress review sessions (January and March) and have training specialist look over your portfolios, answer questions, and share ideas
Make sure you are using current Educator’s Manual, Resource Guides, and forms
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education