Universal Screening At Tier 1

Universal Screening At Tier 1
Mississippi Department of Education 2008

Agenda What is Universal Screening? Essential Element 1 & 2
What is CBM? How to administer CBM Early Literacy Oral Reading Fluency Comprehension Early Numeracy Math Computation How to Universally Screen Essential Element 3 Essential Element 4

What is Universal Screening?
An assessment of all students to identify areas for changes in instructional practices or students who are not proficient relative to specified objectives, indicating that they are at risk for potential difficulties in reading, mathematics (i.e., 1-5 minutes per student) , content subject areas, and/or behavior.

Essential Elements Matrix: Tier 1
1. Universal screening of early literacy skills, oral reading fluency, vocabulary, and reading comprehension, if applicable. Conducts screening of all students (K-8); 3-4 times per year; at approximate equal intervals (i.e., fall, winter, spring); the screening measures the same skill(s) each time with exception to K-1; and measures all applicable skills. Graphical display

Tier 1: Universal Screening
What: A process to collect data on academic and behavioral expectations How: Collect data 3 times a year to demonstrate a developmental trend called “norming” or “universal screening.” Using measures that are quick, brief, and easy to use including Curriculum Based Measurement (CBM) Using content area practice tests Using Office Discipline Referrals or other measures

Overview Why CBM? What is CBM? How do you use it to make decisions?
Definition, differences, etc. Research base How do you use it to make decisions? How do you administer CBM? Early Literacy Reading Math 6/2006

What is CBM? CBM is an established procedure and useful method for determining standards for achievement. Standardized measurement methodology using materials from the curriculum. CBM uses brief (i.e., 1 to 5 min) direct measures of student performance in the basic skill areas of early literacy, reading, comprehension, early numeracy, and math. 6/2006

DESCRIPTION OF CBM Standardized assessment from which you can develop local criteria Dynamic (sensitive) Indicator (correlates) of Basic Skills (not content areas) Formative evaluation Use in a problem solving model Use at individual, class, and school levels 6/2006

Curriculum Based Measurement
“Educational Thermometer” student’s needs can be met before a significant discrepancy occurs

Treatment Utility Use in informing instructional decisions
Math (Fuchs, Fuchs, 1989) Reading (Good, R.H. and Kaminski R.A., 1996). Use in identifying students at risk of academic difficulties

Dynamic Indicators of Basic Early Literacy Skills (DIBELS)
Developed by Richard Good III, Ph.D. Published by Sopris West Licensed by the University of Oregon

What is DIBELS? DIBELS is a collection of brief fluency measures of critical early literacy skills used for: Early identification of students at risk Evaluation of effectiveness of instruction

6/2006

D I B E L S Measures Initial Sound Fluency (PA)
Phoneme Segmentation (PA) Nonsense Word Fluency (P) Letter Naming Fluency Oral Reading Fluency (F) Oral Retelling Fluency (RC) Word Use Fluency (V)

DIBELS Kinder. First Second Third Initial Sound Letter Naming
Oral Reading Oral reading Phoneme Segmentation Nonsense Word

Administration and Scoring

Standardization Essentials
Conduct standardized administration. Every child gets the same opportunity, every time. Give directions verbatim, without elaboration. Time carefully. Use standard scoring system.

Stopwatch Primer Use non-dominant hand (you will be writing with your dominant hand). Click right side of your stopwatch to start and stop. Click left side of your stopwatch to reset stopwatch to 00:00. Stopwatches tell time in seconds and hundredths of seconds, therefore 23:57 seconds can be rounded to 24 seconds.

Initial Sound Fluency Measures the child’s ability to select words starting with a target sound and produce the initial sounds of a word in isolation

Initial Sound Fluency Phonological Awareness
Intended for most children from late preschool through winter of kindergarten Benchmark goal = in mid kindergarten Below 10 in mid K indicates need for intensive instructional support Can be used with older children with low skills in phonological awareness

Directions for Administration
1. Place the student copy of 4 randomized pictures in front of the child. 2. Place the examiner score sheet in front of you, but shielded so that the student cannot see what you record. 3. Say these specific directions to the child: © 2002 Good & Kaminski

Demo This is mouse, flowers, pillow, letters (point to each picture while saying its name). Mouse begins with the sound /m/ (point to the mouse). Listen: /m/, mouse. Which one begins with the sounds /fl/? © 2002 Good & Kaminski 6/2006

Timing Directions 4. Present the first question as written on the score sheet. After you finish asking the question, begin your stopwatch. Stop your stopwatch as soon as the child responds. If the child does not respond after 5 seconds, score the question as zero and present the next question. 5. As soon as the student responds, present the next question promptly and clearly. Begin your stopwatch after you have said the question, and stop it as soon as the student responds, as above. © 2002 Good & Kaminski

Scoring Directions Score the child’s response as either correct (1 point) or incorrect (0 points). If the child stops or struggles with a question for 5 seconds, score the question as zero and present the next question. After the first 4 questions, proceed to the next picture probe. Continue until the end of the questions. When the child finishes the last question, record the total time on your stopwatch in seconds and add the number of correct responses. Record the total number correct and the time in seconds on the bottom of the scoring sheet. © 2002 Good & Kaminski

Letter Naming Fluency (rapid letter naming)
Letter naming fluency works well as a risk indicator for most children from fall of kindergarten through fall of first grade.

LNF Probes Each probe is a random sort of 2 lower case and 2 upper case alphabets. Match font to reading curriculum. Lines help students to keep their place. Serial naming and fluency aspects of the task are important. © Good & Kaminski

Directions 1. Place the student copy of randomized alphabets in front of the student. 2. Place the examiner copy of randomized alphabets in front of you , but shielded so that the student cannot see what you record. 3. Say these specific directions to the student: Here are some letters (point). Tell me the names of as many letters as you can. When I say “begin”, start here (point to first letter), and go across the page (point). Point to each letter and tell me the name of that letter. If you come to a letter you don’t know I’ll tell it to you. Put your finger on the first letter. Ready, begin. © Good & Kaminski

Directions 4. Say "Start" and begin your stop watch. If the student fails to say the first letter after 3 seconds, tell him/her the letter and mark it as incorrect. 5. Follow along on your copy. Put a slash (/) through letters named incorrectly (see scoring procedures) © Good & Kaminski

Directions 6. If a student stops or struggles with a letter for 3 seconds, tell the student the letter and mark it as incorrect. 7. At the end of 1 minute, place a bracket (]) after the last letter named and say, "Stop" 36 © Good & Kaminski

Phoneme Segmentation Fluency (segment each individual sound (PHONEME) in a word) Measures Phonological Awareness Intended for most children from winter of kindergarten through spring of first grade. It may be appropriate for monitoring the progress of older children with low skills in phonological awareness.

Verbal Directions "OK. Here is your first word."
1. Place examiner probe on clipboard and position so that student cannot see what you record. 2. Say these specific directions to the student: I am going to say a word. After I say it, you tell me all the sounds in the word. So, if I say, “Sam,” you would say /s/ /a/ /m/. Let’s try one. (one second pause) Tell me the sounds in “mop”. "OK. Here is your first word." © Good & Kaminski

Procedures 3. Give the student the first word and start your stopwatch. If the student does not say a sound segment after 3 seconds, give him/her the second word and score the first word as zero segments produced. 4. As the student says the sounds, mark the student response in the scoring column. Underline each sound segment produced correctly. Put a slash (/ ) through sounds produced incorrectly. © Good & Kaminski

Time Limit 5. As soon as the student is finished saying the sounds, present the next word promptly and clearly. 6. The maximum time for each sound segment is 3 seconds. If the student does not provide the next sound segment within 3 seconds, give the student the next word. If student provides the initial sound only, wait 3 seconds for elaboration. 7. At the end of 1 minute, stop presenting words and scoring further responses. Add the number of sound segments produced correctly. Record the total number of sound segments produced correctly on the bottom of the scoring sheet. © Good & Kaminski

Scoring Rules 1. Discontinue Rule. If a student has not given any sound segments correctly in the first 5 words, discontinue the task and put a score of zero (0). 2. Underline the sound segments in the word the student produces that are correctly pronounced. Students receive 1 point for each different, correct, part of the word. 3. Put a slash ( ) through segments pronounced incorrectly. © Good & Kaminski

Measures Alphabetic Principles
Nonsense Word Fluency (decoding make-believe words) Measures Alphabetic Principles *Nonsense Word Fluency is intended for most children from spring of kindergarten through spring of first grade. It may be appropriate for monitoring the progress of older children with low alphabetic principle and decoding skills.

Administration Directions
Practice Items Look at this word (point to the first word on the practice probe). It’s a make-believe word. Watch me read the word: /s/ /i/ /m/ “sim” (point to each letter then run your finger fast beneath the whole word). I can say the sounds of the letters, /s/ /i/ /m/ (point to each letter), or I can read the whole word “sim” (run your finger fast beneath the whole word). sim lut Your turn to read a make-believe word. Read this word the best you can (point to the word “lut”). Make sure you say any sounds you know. © Good & Kaminski

Student Copy 4. Place the student copy of the probe in front of the child. Here are some more make-believe words (point to the student probe). Start here (point to the first word) and go across the page (point across the page). When I say, “begin”, read the words the best you can. Point to each letter and tell me the sound or read the whole word. Read the words the best you can. Put your finger on the first word. Ready, begin. 5. Start your stopwatch. © Good & Kaminski

Examiner Copy Benchmark 1 Nonsense Word Fluency 6. Follow along on the examiner copy of the probe and underline each phoneme the student provides correctly, either in isolation or in the context of the nonsense word. Put a slash (/) over each phoneme read incorrectly or omitted. Error Pattern: © Good & Kaminski

Administer for 1 Minute 7. At the end of 1 minute, place a bracket (]) after the last letter sound provided by the student and say, “Stop.” 8. These directions can be shortened by beginning with Number 4 for repeated measurement when the student clearly understands the directions and procedure. Directions for Scoring 1. Discontinue Rule. If the student does not get any sounds correct in words 1-5, discontinue the task and record a score of 0. © Good & Kaminski

Correct letter sounds. 2. Correct letter sounds. Underline the individual letters for letter sounds produced correctly in isolation and score 1 point for each letter sound produced correctly. For example, if the stimulus word is “tob” and the student says /t/ /o/ /b/, the individual letters would be underlined with a score of 3. © Good & Kaminski

Correct Words 3. Correct words. Use a single underline under multiple letters for correct letter sounds blended together and give credit for each letter sound correspondence produced correctly. For example, if the stimulus word is “tob” and the student says “tob”, one underline would be used with a score of 3. © Good & Kaminski

Partially correct words
4. Partially correct words. If a word is partially correct, underline the corresponding letters for letter sounds produced correctly. Put a slash ( ) through the letter if the corresponding letter sound is incorrect. For example, if the word is “tob” and the student says “toab” (with a long o), the letters “t” and “b” would be underlined, and the letter “o” would be slashed with a score of 2. 6/2006 © Good & Kaminski

DIBELS Oral Reading Fluency Measure (DORF)
Intended for most children from mid first grade through third grade Measures accuracy and fluency with connected text Used to: Identify children who may need additional instructional support Monitor progress toward instructional goals

Automaticity with the Code
What is it? The ability to quickly and accurately apply letter-sound correspondence to reading connected text. Automaticity provides an overall indicator of student growth and development in early reading skills. What it is not? Oral reading fluency will not tell you everything you need to know about student reading performance. However, there is a strong relationship between oral reading fluency and comprehension.

Directions for Administration
1. Place the reading passage in front of the student. 2. Place the examiner copy on clipboard and position so that the student cannot see what you record. © Good & Kaminski

Directions 3. Say these specific directions to the student:
Please read this (point) out loud. If you get stuck, I will tell you the word so you can keep reading. When I say, “stop” I may ask you to tell me about what you read, so do your best reading. Start here (point to the first word of the passage). Begin. © 2001 Good & Kaminski Page 6 © Good & Kaminski

Timing 4. Start your stopwatch when the student says the first word of the passage. The title is not counted. If the student fails to say the first word after 3 seconds, tell them the word and mark it as incorrect, then start your stopwatch. 5. The maximum time for each word is 3 seconds. If the student does not provide the word within 3 seconds, say the word and mark the word as incorrect. 6. Follow along on the examiner copy of the probe. Put a slash ( ) over words read incorrectly. © Good & Kaminski

Scoring 7. At the end of 1 minute, place a bracket ( ] ) after the last word provided by the student and say “Stop” and remove the passage. © Good & Kaminski

Directions for Scoring
1. Score reading passages immediately after administration. 2. Discontinue Rule. If the student does not read any words correctly in the first row, discontinue the task and record a score of 0. 3. Record the total number of words read correctly on the bottom of the scoring sheet for each passage. 4. If the student reads fewer than 10 words correct on the first passage, record their score on the front cover and do not administer passages 2 and 3. 5. If the student reads 3 passages, record their middle score on the front cover. For example, if the student gets scores of 27, 36, and 25, record a score of 27 on the front cover. © Good & Kaminski

Reading Administration Instructions Scoring
Individual (one adult : one child) Three 1 min probes; estimated 5 min per child Instructions Standardized for all grades Scoring Correct Words Per Minute (WCPM) Number of words counted on right-hand side Accuracy: count number of correct words Read aloud for only one minute Is usually growth in number of words read aloud 6/2006

Reading - Curriculum Based Measurement (R-CBM)
Play Reading Clip 1

Observation Questions
What did you observe about this child’s reading? Is she a good reader? Give your reason(s) for your answer to the second question? About how many words did she read correctly?

Reading - Curriculum Based Measurement (R-CBM)
Play Reading Clip 2

What did you observe about this child’s reading? Is she a good reader? Give your reason(s) for your answer to the second question? About how many words did she read correctly?

Administration and Scoring of R-CBM
What Examiners Need to Do . . . Before testing students While testing students After testing students

Things You Need Before Testing
Standard Reading Assessment Passage Student Copy: No numbers Between words (exception: 1st grade) An informative first sentence Same font style and size Text without pictures

Additional Assessment Aids
A List of Students to be Tested Stop Watch (required) Clipboard Pencil

Setting up Assessment Environment
Assessment environments are flexible and could include… A set-aside place in the classroom A reading station in the hallway Reading stations in the media center, cafeteria, gym, or empty classrooms

Things You Need to do While Testing
Follow the standardized directions: R-CBM is a standardized test Administer the assessment with consistency Remember it’s about testing, not teaching Don’t teach or correct Don’t practice reading the passages Remember Best, not fastest reading Sit across from, not besides, the student

Things You Need Before Testing (Continued)
Standard Reading Assessment Passage Examiner Copy: Pre-numbered so they can be scored quickly and immediately. AIMSweb 6/2006

R-CBM Standard Directions for 1 Minute Administration
Place the unnumbered copy in front of the student. Place the numbered copy in front of you, but shielded so the student cannot see what you record. Say: When I say ‘Begin,’ start reading aloud at the top of this page. Read across the page (DEMONSTRATE BY POINTING). Try to read each word. If you come to a word you don’t know, I will tell it to you. Be sure to do your best reading. Are there any questions? (PAUSE) Say “Begin” and start your stopwatch when the student says the first word. If the student fails to say the first word of the passage after 3 seconds, tell them the word, mark it as incorrect, then start your stopwatch. Follow along on your copy. Put a slash ( / ) through words read incorrectly. At the end of 1 minute, place a bracket ( ] ) after the last word and say, “Stop.” Score and summarize by writing WRC/Errors AIMSweb 6/2006

Reading (example from SBG, 1989)
Name_______ TWR ______ Date: _______ Errors ______ WRC ______ Once, while traveling along the seashore, I stopped at a small fishing village. After eating lunch, I decided to take a walk. I followed a path out of the village, uphill to some cliffs high above the sea. At the edge of these cliffs was a most unusual sight - the wreck of a small sailboat. An old man was sitting among the broken timbers, smoking a pipe. He seemed to be reading my mind when he said, “Odd, isn’t it?’ 7 15 25 34 44 54 55 62 70 79 80

Reading (example from SBG, 1989)
Name_Cindy_ TWR _ 69___ Date: _9/21/ Errors __5___ WRC _64___ Once, while traveling along the seashore, I stopped at a small fishing village. After eating lunch, I decided to take a walk. I followed a path out of the village, uphill to some cliffs high above the sea. At the edge of these cliffs was a most unusual sight - the wreck of a small sailboat. An old man was sitting among the broken timbers, smoking a pipe. He seemed to be reading my mind when he said, “Odd, isn’t it?’ 7 15 25 34 44 54 55 62 70 79 80

“Familiar” Shortened Directions
When students are assessed frequently and know the directions. Say: When I say ‘Begin,’ start reading aloud at the top of this page. 6/2006

Items to Remember Emphasize Words Read Correctly (WRC). Get an accurate count. 3-Second Rule No other corrections Discontinue rule Be polite Best, not fastest Interruptions 6/2006

Accuracy of Implementation (AIRS)
AIMSweb 6/2006

Things to Do After Testing
Score immediately to ensure accurate results! Determine WRC Put a slash (/) through incorrect words If doing multiple samples, organize your impressions of qualitative features 6/2006

What is a Word Read Correctly?
Correctly pronounced words within context Self-Corrected Incorrect Words within 3 seconds 6/2006

What is an Error? Mispronunciation of the Word or Substitutions Omissions Stops or struggles with a letter for more than 3 seconds (examiner provides correct word) 6/2006

What is not Incorrect? (Neither a WRC or an Error)
Repetitions Dialect Differences Insertions (Consider them Qualitative Errors) 6/2006

Calculating and Reporting Scores
Count the total number of words the student read Count the number of errors and subtract Report in standard format of WRC/Errors (72/3) 6/2006

Example of Calculating Scores
Juan finished reading after 1 minute at the 145th word, so he read 145 words total Juan also made 3 errors Therefore, his WRC was 142 with 3 errors Reported as 142/3 6/2006

Determining Inter-Rater Agreement
Agreements/(Agreements + Disagreements) x 100 Example: Examiner 1 = 100 WRC Examiner 2 = 98 WRC Inter-rater agreement = 98% 6/2006

Computation They agreed that the student read 98 of the words correct.
They disagreed on 2 words correct. Agreements (98)/(Agreements + Disagreements) x 100 98/(98 + 2)= 98/100 = .98 .98 x 100 = 98% 6/2006

Practice Exercise 1: Let’s Score
Play Reading PE 1

Practice Exercise 1: Answer Key
This student read 72 WRC/8 Errors 6/2006

Play Reading PE 2 6/2006

Play Reading PE 3 6/2006

Curriculum Based Measurement Reading Maze (Continued)
Maze is a multiple-choice cloze task that students complete while reading silently. The students are presented with word passages. The first sentence is left intact. After the first sentence, every 7th word is replaced with three word choices inside a parenthesis. The three choices consist of, Near Distracter Exact Match Far Distracter

An example of CBM Maze

What did you observe about Emma’s and Abby’s Maze performance? What other conclusions can you draw?

Items Students Need Before Testing
What the Students Need for Testing: CBM Maze practice test (optional) Appropriate CBM Maze passages Pencils

Items Administrators Need Before Testing
What the Tester Uses for Testing: Stopwatch Appropriate CBM Maze Answer Key Appropriate Standardized Directions

Additional Assessment Aids
A List of Students to be Tested Stop Watch (required)

Setting up Assessment Environment
Assessment environments are flexible and could include… The classroom if assessing the entire class. A cluster of desks or small tables in the classroom for small group assessment. Individual desks or “stations” for individual assessment.

Things You Need to do While Testing
Follow the Standardized Directions Attach a cover sheet that includes the practice test so that students do not begin the test right away. Do a simple practice test with younger students. Monitor to ensure students are circling answers instead of writing them. Be prepared to “Prorate” for students who may finish early. Try to avoid answering student questions. Adhere to the End of Timing.

CBM Maze Standard Directions
Pass Maze tasks out to students. Have students write their names on the cover sheet, so they do not start early. Make sure they do not turn the page until you tell them to. Say this to the student (s): When I say ‘Begin’ I want you to silently read a story. You will have 3 minutes to read the story and complete the task. Listen carefully to the directions. Some of the words in the story are replaced with a group of 3 words. Your job is to circle the 1 word that makes the most sense in the story. Only 1 word is correct. Decide if a practice test is needed. Say . . . Let’s practice one together. Look at your first page. Read the first sentence silently while I read it out loud: ‘The dog, apple, broke, ran after the cat.’ The three choices are apple, broke, ran. ‘The dog apple after the cat.’ That sentence does not make sense. ‘The dog broke after the cat.’ That sentence does not make sense. ‘The dog ran after the cat.’ That sentence does make sense, so circle the word ran. (Make sure the students circle the word ran.)

CBM Maze Standard Directions (Continued)
Let’s go to the next sentence. Read it silently while I read it out loud. ‘The cat ran fast, green, for up the hill. The three choices are fast, green, for up the hill. Which word is the correct word for the sentence? (The students answer fast) Yes, ‘The cat ran fast up the hill’ is correct, so circle the correct word fast. (Make sure students circle fast) Silently read the next sentence and raise your hand when you think you know the answer. (Make sure students know the correct word. Read the sentence with the correct answer) That’s right. ‘The dog barked at the cat’ is correct. Now what do you do when you choose the correct word? (Students answer ‘Circle it’. Make sure the students understand the task) That’s correct, you circle it. I think you’re ready to work on a story on your own.

CBM Maze Standard Directions (Continued)
Start the testing by saying . . . When I say ‘Begin’ turn to the first story and start reading silently. When you come to a group of three words, circle the 1 word that makes the most sense. Work as quickly as you can without making mistakes. If you finish a/ the page/first side, turn the page and keep working until I say ‘Stop’ or you are all done. Do you have any questions? Then say, ‘Begin.’ Start your stopwatch. Monitor students to make sure they understand that they are to circle only 1 word. If a student finished before the time limit, collect the student’s Maze task and record the time on the student’s test booklet. At the end of 3 minutes say: Stop. Put your pencils down. Please close your booklet. Collect the Maze tasks.

CBM Maze Familiar Directions
After the students have put their names on the cover sheer, start the testing by saying . . . When I say ‘Begin’ turn to the first story and start reading silently. When you come to a group of three words, circle the 1 word that makes the most sense. Work as quickly as you can without making mistakes. If you finish a/ the page/first side, turn the page and keep working until I say ‘Stop’ or you are all done. Do you have any questions? Then say, ‘Begin.’ Start your stopwatch. Monitor students to make sure they understand that they are to circle only 1 word. If a student finished before the time limit, collect the student’s Maze task and record the time on the student’s test booklet. At the end of 3 minutes say: Stop. Put your pencils down. Please close your booklet. Collect the Maze tasks.

Things to Do After Testing
Score immediately to ensure accurate results! Determine the Number of Words (Items) Correct. Use the answer key and Put a slash (/) through incorrect words

An answer is considered an error if the student:
CBM Maze Scoring What is correct? The students circles the word that matches the correct word on the scoring template. What is incorrect? An answer is considered an error if the student: Circles an incorrect word Omits word selections other than those the student was unable to complete before the 3 minutes expired

Making Scoring Efficient
Count the total number of items up to the last circled word. Compare the student answers to the correct answers on the scoring template. Mark a slash [/] through incorrect responses. Subtract the number of incorrect answers from the total number of items attempted. Record the total number of correct answers on the cover sheet followed by the total number of errors (e.g., 35/2).

Convert the time taken in seconds. (2 minutes = 120 seconds)
CBM Maze Prorating If a student finishes all the items before 3 minutes, the score can be prorated . When the student finished must be recorded and the number of correct answers counted. For example, the student may have finished in 2 minutes and correctly answered 40 items. Convert the time taken in seconds. (2 minutes = 120 seconds) Divide the number of seconds by the number correct. (120/40 = 3) Calculate the number of seconds in the full 3 minutes. (3 minutes = 180 seconds) Divide the number of full seconds by the calculated value from step 3. (180/3 = 60)

Summary You now have the building blocks to begin CBM Maze to ensure literacy growth. Practice to Automaticity --You’ll get more efficient Get Checked Out with AIRS for accuracy/efficiency by a colleague Stay in Tune by periodically checking AIRS

2. Universal screening of early numeracy skills, math fluency, and reasoning, if applicable Conducts screening of all students (K-8); 3-4 times per year; at approximate equal intervals (i.e., fall, winter, spring); the screening measures the same skill(s) each time with exception to K-1; and measures all applicable skills. Graphical display

Early Numeracy Measures
Students perform different early numeracy tasks related to later important math outcomes. Measure Test Arrangement What is Scored Oral Counting (OC) Individual Correct Oral Counts Number Identification (NI) Correct Number Identifications Quantity Discrimination (QD) Correct Quantity Discriminations Missing Number (MN) Correct Missing Numbers

Recommended Early Numeracy Benchmark Assessment Schedule
Kindergarten First Grade Fall Winter Spring Oral Counting Number Identification Number Discrimination Missing Number

Getting the Necessary Testing Materials
Benefits of Using Standard Early Numeracy Assessment Probes Are written to represent general curriculum or be “curriculum independent.” Allow decision making about early numeracy growth, regardless of between-school, between-district, and between- teacher differences in math curriculum. Are graded to be of equal difficulty. Have numerous alternate forms for testing over time without practice effects.

Arranging the Testing Environment
Helpful Hints While Administering Early Numeracy Measures Make sure that the testing environment is quiet and free from distractions Complete reliability checks before data collection with all examiners to ensure reliable administration If possible, try to have the same examiner during each Benchmark period test the same students

Administration and Scoring of Oral Counting

Administration and Scoring of Early Numeracy Oral Counting (OC)

Things You Need Before Testing Oral Counting (OC)
Specific Materials Arranged: Examiner copy of Oral Counting for scoring Clipboard to provide a hard surface for recording student answers Stopwatch Tape recorder (optional)

Things You Need While Testing Oral Counting
Standard Directions for 1- Minute Administration Place the examiner copy on a clipboard and position so the student cannot see what the examiner records. Say these specific directions to the student: “When I say start I want you to start counting aloud from 1 like this 1, 2, 3 until I tell you to stop. If you come to a number you don’t know, I’ll tell it to you. Be sure to do your best counting. Art there any questions? Ready, Start.”.” Start you stopwatch. If the student fails to say “1” after 3 seconds, say “1” and continue. Follow along on the examiner copy. Score according to scoring rules. After one minute has expired, place a bracket after the last number said and say “Stop.”

Things to Do After Testing Oral Counting (OC)
Score immediately to ensure accurate results Students receive 1 point for every oral count in 1 minute If doing multiple samples, organize your impressions of qualitative features

Oral Counting Scoring Rules
Rule 1: If a student says the number that comes next in their sequence, score the number as correct. Example: Student says , 13, Scoring is 3 correct. Rule 2: If a student repeats a sequence, score the sequence as correct. Example: Student says 7, 8, 7, 8, 9. Scoring is 3 correct. Rule 3: If a student self-corrects, score the number as correct. Example: Student says 1, 2, 3, 5 oh 3, 4, 5. Scoring is 5 correct

Oral Counting Scoring Rules (Continued)
Rule 4: If a student hesitates with a number for 3 seconds, tell the student the number and score the number on which the student hesitated as incorrect. You may tell the student only one number during testing. Example: Student says 12, 13, .…(3 seconds) examiner says “14”. Score 14 as incorrect and continue testing. Student continues to count and comes to 28, 29, (3 seconds) examiner says "stop” and discontinues testing. Rule 5: If a student skips a number while counting, score the number as incorrect and continue testing. Example: Student says 6, 7, 9. Score 8 as incorrect and continue testing.

Calculating and Reporting Scores Oral Counting
Count the total number of numbers the student counted Count the number of errors and subtract Report the total number of correct oral counts

Exercise 1 Early Numeracy Oral Counting: Let’s Practice!
Play OC PE 1

Practice Exercise 1 Early Numeracy Oral Counting: Answer Key

Exercise 2 Early Numeracy Oral Counting: Let’s Practice!
Play OC PE 2

Practice Exercise 2 Early Numeracy Oral Counting: Answer Key

Determining Inter-Rater Agreement
Agreements/(Agreements + Disagreements) x 100 Dave Examiner 1 = 24 items correct Examiner 2 = 25 items correct Inter-rater agreement = 96%

Administration and Scoring of Number Identification

What Examiners Need to Do . . .
Administration and Scoring of Early Numeracy Number Identification (NI) What Examiners Need to Do . . . Before testing students While testing students After testing students

Things You Need Before Testing Number Identification (NI)
Specific Materials Arranged: Examiner copy of Number Identification for scoring Student copy of Number Identification for Scoring Clipboard to provide a hard surface for recording student answers Stopwatch Tape recorder (optional)

Things You Need While Testing Number Identification
Standard Directions for 1- Minute Administration Place the student copy in front of the student. Place the examiner copy on a clipboard so the student cannot see what the examiner records. Say these specific directions to the student: “Look at the paper in front of you. It has a number on it (demonstrate by pointing). What number is this.” Example 1 Correct Response “Good. The number is 8. Look at the number next to 8 (demonstrate by pointing). What number is this?” Incorrect Response “This number is 8 (point to 8). What number is this? Good. Let’s try another one. Look at the number next to 8 (demonstrate by pointing). What number is this?” Workbook p. 16

Things You Need While Testing Number Identification (Continued)
Example 2 Correct Response “Good. The number is 4.” (Turn the page). Incorrect Response “This number is 4 (point to 4). What number is this? Good.” (Turn the page). “The paper in front of you has numbers on it. When I say start, I want you to tell me what the numbers are. Start here and go across the page (demonstrate by pointing). If you come to a number you don’t know, I’ll tell you what to do. Are there any questions? Put your finger on the first one. Ready, start.”

Things You Need While Testing Number Identification (Continued)
Start your stopwatch. If the student fails to answer the first problem after 3 seconds, tell the student to “try the next one.” If the student does not get any correct within the first 5 items, discontinue the task and record a score of zero. Follow along on the examiner copy. Put a slash (/) through any incorrects. The maximum time for each item is 3 seconds. If a student does not provide an answer within 3 seconds, tell the student to “try the next one.” At the end of 1 minute, place a bracket (]) around the last item completed and say “Stop.”

Things to Do After Testing Number Identification (NI)
Score immediately to ensure accurate results Students receive 1 point for every item correctly completed in 1 minute If doing multiple samples, organize your impressions of qualitative features

Number Identification Scoring Rules
Rule 1: If a student correctly identifies the number, score the item as correct. Rule 2: If the student states any number other than the item number, score the item as incorrect. Rule 3: If the student hesitates or struggles with a number for 3 seconds, tell the student to “try the next one” and score the item as incorrect.

Number Identification Scoring Rules (Continued)
Rule 4: If a student skips a number, score the number as incorrect. Rule 5: If a student skips an entire row, mark each number in the row as incorrect by drawing a line through the row on the examiner score sheet. Rule 6: If the student misses 5 items consecutively discontinue testing.

Exercise 3 Early Numeracy Number Identification: Let’s Practice!
Play NI PE 3

Practice Exercise 3 Early Numeracy Number Identification: Answer Key

Exercise 4 Early Numeracy Number Identification: Let’s Practice!
Play NI PE 4

Practice Exercise 4 Early Numeracy Number Identification: Answer Key

Administration and Scoring of Quantity Discrimination

What Examiners Need to Do . . .
Administration and Scoring of Early Numeracy Quantity Discrimination (QD) What Examiners Need to Do . . . Before testing students While testing students After testing students

Things You Need Before Testing Quantity Discrimination (QD)
Specific Materials Arranged: Examiner copy of Quantity Discrimination for scoring Student copy of Quantity Discrimination for Scoring Clipboard to provide a hard surface for recording student answers Stopwatch Tape recorder (optional)

Things You Need While Testing Quantity Discrimination
Standard Directions for 1- Minute Administration Place the student copy in front of the student. Place the examiner copy on a clipboard and position so the student cannot see what the examiner records. Say these specific directions to the student: “Look at the piece of paper in front of you. The box in front of you has two numbers in it (demonstrate by pointing). I want you to tell me the number that is bigger.” Example 1 Correct Response “Good. The bigger number is 7. Now look at this box (demonstrate by pointing). It has two numbers in it. Tell me the number that is bigger.” Incorrect Response “The bigger number is 7. You should have said 7 because 7 is bigger than 4. Now look at this box (demonstrate by pointing). It has two numbers in it. Tell me the number that is bigger.” Workbook p. 16

Things You Need While Testing Quantity Discrimination (Continued)
Example 2 Correct Response “Good. The bigger number is 4.” (Turn the page). Incorrect Response “The bigger number is 4 . You should have said 4 because 4 is bigger than 2. (Turn the page). “The paper in front of you has boxes on it. In the boxes are two numbers. When I say start, I want you to tell me the number in the box that is bigger. Start here and go across the page (demonstrate by pointing). If you come to a box and you don’t know which number is bigger, I’ll tell you what to do. Are there any questions? Put your finger on the first one. Ready, start.”

Things You Need While Testing Quantity Discrimination (Continued)

Things to Do After Testing Quantity Discrimination (DQ)

Quantity Discrimination Scoring Rules
Rule 1: If a student states the bigger number, score the item as correct. Rule 2: If the student states both numbers, score the item as incorrect. Rule 3: If the student states any number other than the bigger number, score the item as incorrect.

Quantity Discrimination Scoring Rules (Continued)
Rule 4: If a student hesitates or struggles with an item for 3 seconds, tell the student to “try the next one.” Score the item as incorrect. Rule 5: If a student skips an item, score the item as incorrect. Rule 6: If a student skips an entire row, mark each item in the row as incorrect by drawing a line through the row on the examiner score sheet. Rule 7: If a student misses 5 items consecutively, discontinue testing.

Exercise 5 Early Numeracy Quantity Discrimination: Let’s Practice!
Play QD PE 5

Practice Exercise 5 Early Numeracy Quantity Discrimination: Answer Key
Workbook p. 18

Exercise 6 Early Numeracy Quantity Discrimination: Let’s Practice!
Play QD PE 6

Practice Exercise 6 Early Numeracy Quantity Discrimination: Answer Key

Administration and Scoring of Missing Number

Administration and Scoring of Early Numeracy Missing Number (MN)

Things You Need Before Testing Missing Number (MN)
Specific Materials Arranged: Examiner copy of Missing Number for scoring Student copy of Missing Number for Scoring Clipboard to provide a hard surface for recording student answers Stopwatch Tape recorder (optional)

Things You Need While Testing Missing Number
Standard Directions for 1- Minute Administration Place the student copy in front of the student. Place the examiner copy on a clipboard and position so the student cannot see what the examiner records. Say these specific directions to the student: “The box in front of you has two numbers in it (point to the first box). I want you to tell me the number that goes in the blank. What number goes in the blank?.” Example 1 Correct Response “Good. 1 is the number that goes in the blank Let’s try another one (point to the second box). What number goes in the blank?” Incorrect Response “The number that goes in the box is 1. See 1, 2, 3 (demonstrate by pointing). 1 goes in the blank. Let’s try another one (point to the second box). What number goes in the blank?” Workbook p. 16

Things You Need While Testing Missing Number (Continued)
Example 2 Correct Response “Good. 7 is the number that goes in the blank.” (Turn the page). Incorrect Response “The number that goes in the blank is 7. See 5, 6, 7 (demonstrate by pointing). 7 goes in the blank.” (Turn the page). “The piece of paper in front of you has boxes with numbers in them. When I say start, you are going to tell me the number that goes in the blank for each box. Start with the first box and go across the row (demonstrate by pointing). If you come to one you don’t know, I’ll tell you what to do. Are there any questions? Put your finger on the first one. Ready, start.”

Things You Need While Testing Missing Number (Continued)

Things to Do After Testing Missing Number (MN)

Missing Number Scoring Rules
Rule 1: If a student correctly states the missing number, score the item as correct. Rule 2: If a student incorrectly states the missing number next, score the item as incorrect by placing a slash through the number on the examiner score sheet. Rule 3: If a student hesitates or struggles with an item for 3 seconds, tell the student to “try the next one” (demonstrate by pointing). Score the item as incorrect.

Missing Number Scoring Rules (Continued)
Rule 4: If a student skips an item, score the item as incorrect. Rule 5: If a student skips an entire row, mark each item in the row as incorrect by drawing a linethrough the row on the examiner score sheet. Rule 6: If a student misses 5 items consecutively, discontinue testing.

Exercise 7 Early Numeracy Missing Number: Let’s Practice!
Play MN PE 7

Practice Exercise 7 Early Numeracy Missing Number: Answer Key

Exercise 8 Early Numeracy Missing Number: Let’s Practice!
Play MN PE 8

Practice Exercise 8 Early Numeracy Missing Number: Answer Key

Early Numeracy Measures Reviewed:
Summary You now have the building blocks to begin AIMSweb Early Numeracy Assessment to ensure mathematic growth. Early Numeracy Measures Reviewed: Oral Counting (OC) Number Identification (NI) Quantity Discrimination (QD) Missing Number (MN)

MATH PROBE Variety of types of problems the student will encounter
Group administration Three to five minute test “Correct digits” is the number of digits in the correct place on each problem 6/2006

M-CBM Methodology: Based on expected math computation skills for Grades 1-6: Benchmarking: M-CBM (10 probes available per grade level) Strategic Monitoring: M-CBM (40 probes available per grade level) Progress Monitoring: M-CBM (40 probes available per grade level) or: Progress Monitoring: Math Facts: (40 probes available per grade level) Each probe contains 2 pages of computations, (front/back) Students complete probes under standardized conditions (See specific directions) Administer individually, small group, or class-wide: (Grade 1-3 = 2 minutes) (Grade 4-6 = 4 minutes)

M-CBM Grade-Level Probes
All M-CBM Grade-level type probes level are based on Prototype Probe Prototype Probes: All types of problems contained within the scope of a grade-level skill spectrum are consistent in order/sequence across all probes for each specified grade level. A-1: Same type of problem on each page Grade 2: Probe 1 Grade 2: Probe 2

M-CBM Sample 2nd Grade Probe (Student Copy)

M-CBM Sample 2nd Grade Probe (Teacher’s Answer Key)
Scoring based on number of DIGITS CORRECT (DS)

M-CBM Math Facts: Addition
Page 1 of 2 Page 1 of 2 Grades 1-3 (fewer problems) Grades 4-6 (more problems)

M-CBM Math Facts: Probe options
Addition (+) facts Subtraction (-) facts Addition/Subtraction (+ / -) mixed facts Multiplication facts (x) Division facts (÷) Multiplication/Division mixed facts (x / ÷) Addition/Subtraction/Multiplication/Division mixed facts (+, -, x, ÷)

M-CBM Administration Setup
Things you will need: Appropriate M-CBM or Math Facts Probes Students need pencils. Stopwatch or timers Setting up the testing room: Large group: Monitor carefully to ensure students are not skipping and X-ing out items Small group/individual: Monitor similarly. If students are off task, cue with statements such as: “Try to do EACH problem.” or “You can do this kind of problem so don’t skip.”

M-CBM Directions for Administration
Different M-CBM’s require different instructions (4 versions): Standard Math-Curriculum Based Measurement Probe instructions: Grades 1-3: Page 13 of workbook Grades 4-6: Page 14 of workbook Single-Skill Math Fact Probes—Standard Directions: Grades 1-6 probes: Page 15 of workbook Multiple Skill Math Fact Probes—Standard Directions: Grades 1-6 probes: Page 16 of workbook

M-CBM After Testing—Scoring
What is Correct? Grades 1-6: Score Digits Correct (DC). Each digit correct in any answer = 1 point. If problem is “X”-ed out, ignore X and score anyway. Use Answer Key for quick scoring. See Page 18 of workbook for examples.

M-CBM: How to score Correct Digits (CD)
Correct Digits - Each correct digit that a student writes is marked with an underline and counted.

M-CBM: How to score Incomplete Problems
Incomplete Problems - Sometimes students don’t finish a problem. Score for the number of correct digits that are written.

M-CBM: How to score “X”-ed out problems
X-ed Out Problems - Sometimes students start a problem and then cross it out. Sometimes students go back and write answers for problems they have crossed out. Ignore the X and score what you see.

M-CBM: How to score reversals
Legibility and Reversed or Rotated Numbers - Sometimes trying to figure out what number the student wrote can be challenging, especially with younger students or older students with mathematics achievement problems. To make scoring efficient and reliable, we recommend attention to three rules: 1. If it is difficult to determine what the number is at all, count it wrong. 2. If the reversed number is obvious, but correct, count it as a correct digit. 3. If the numbers 6 or 9 are potentially rotated and the digit is currently incorrect, count it as an incorrect digit.

How to score M-CBM: Critical Processes (CP) Scoring
Scoring Rules for Answer and Critical Processes: When students’ Grade 5 or Grade 6 M-CBM probes are scored for the number of CDs in the answer only and critical processes, the examiner uses the answer key that details which digits are to be counted. Each problem has an “assigned CD value” based on what AIMSweb believes is the most conventional method of solving the computational problem. Compare how the same multi-step multiplication problem would be scored using the different methods. Answer & Critical Processes: Answer Only: Use error patterns diagnostically

How to score M-CBM: Critical Processes (CP) Scoring (Correct vs
How to score M-CBM: Critical Processes (CP) Scoring (Correct vs. Incorrect) Should the student solve the problem and not write any of the CD, the score would be 0 CD. Should the student solve the problem correctly, their score would be 13 CD. (p. 21 in M-CBM Workbook) Although you don’t need to count every digit written in a correct answer, it is important to write the number of CDs awarded to the problem next to the answer.

How to score M-CBM: Critical Processes (CP) Scoring (Work not shown)
If they do not show their work and the answer is incorrect, the examiner can only “score what they see” as shown below. If the answer is correct as shown below-left, their score is the number of CDs possible shown in the answer key. (p. 21 in M-CBM Workbook)

How to score M-CBM: Answer & Critical Processes (CP) Scoring (Alignment)
If the answer is incorrect, count the digits as they appear in approximate place value as shown below, even if a place value error may seem obvious. When students’ answers are not aligned correctly according to place value: If the answer is correct, ignore the alignment problem and count the digits as correct as shown below. (p. 22 in M-CBM Workbook)

CBM is the vehicle that allows the establishment and implementation of local norms.

Essential Element 3 3. Universal screening for students who will be taking Algebra I, Biology I, U. S. History, and English II. ___Conducts screening of all students (9-12) or grades in your school; ___3-4 times per year; ___at approximate equal intervals (i.e., fall, winter, spring); ___measures all applicable skills. ___ implementation integrity

Essential Element 3 Practice Tests on Content Areas
We are attempting to create, develop, or procure assessments that will lead to better results for this essential element.

Proposing Local Norming to the School District
Meeting with the administrators Clearly outline benefits Describe procedures Results Offer follow-up trainings Offer in-service on results Be available, planful, & visible Get feedback

Establish a Team Research Team District Team Teacher Level
Administrative Level Special Education Level Research Team: Graduate students in a training program, any university close-by Establish Team Proposal to school, IRB process… WRITTEN PLAN! Meet with superintendent, propose project Meet with principals (two members at each meeting) Memos to teachers, Attend faculty meetings, etc. 6. Establish schedule District Team: Teachers; Principals; Sped teachers Easier if district decides to do it.

Scheduling School Schedule Transition Times Teacher Courtesy
Faculty meetings Data Collectors Schedule Estimation of collection of data Prepare for the Worst! Know recess, activities, LUNCH, know class changes Hurricanes, Locking keys in trunk!

Purpose of Local Norms Outcome information Reading Fluency: CWPM
Math Fluency: CDPM Spelling: CLS Writing Fluency Purpose:

Purpose of Local Norms Identify needs Assess skills
Develop interventions

IMPLEMENTATION Support from Stakeholders Develop Timeline
Identify Personnel Resources Staff Training Computer Needs Assessment Responsibilities at all three tiers

Examples of Probe Development or Procurement
Early Literacy Intervention Central DIBELS AIMSWeb iSTEEP Reading AIMSweb ISTEEP Fluency Plus, LLC Mazes Make your own AIMSweb Reading Fluency, LLC Vocabulary Early Numeracy Intervention Central Math Computation School House Technologies

RESOURCES University of Oregon – Dibels http://dibels.uorgon.edu
Florida Project - CBM Website List - Curriculum-Based Measurement – Assessing Special Children by Mark Shinn Advanced Applications of Curriculum Based Measurement by Mark Shinn The CBM Program!

4. Universal screening of behavior Conducts screening of all students (K-12); using the same measure; 3-4 times per year; and at approximate equal intervals (i.e., fall, winter, spring). Graphical display

Types of Data to Collect

Universal Screening of Behavior at Tier I
Office Discipline Referrals (ODR) Systematic Screening for Behavior Disorders (SSBD) Walker, H.M., & Severson, H.H. (1992). Systematic screening for behavior disorders (SSBD): User’s guide and administration manual. Longmont, CO: Sopris West. Behavior Assessment System for Children (BASC) Teacher Report Form – Screener Kamphaus, R.W., Thorpe, J.S., Winsor, A.P., Kroncke, A.P., Dowdy, E.T., & VanDeventer, M. (2007). Development and predictive validity of a teacher screener for child behavioral and emotional problems at school. Educational and Psychological Measurement, 6, 1-15. Could also look at academic issues such as grades, MCT performance, and retentions.

Tier 1: Universal Screening
Behavioral At-Risk Office Discipline Referrals Measure of discipline and classroom management effectiveness Valid measure for school climate Identify students at risk Locate areas to target better Tier I management (e.g., Classroom, hallway, playground, etc)

Universal Screening of Behavior
Office Discipline Referrals Need to examine Code of Conduct Are the rules or expectations clear? Need to examine office discipline referrals sheets Does it have : Date Time Location Teacher Infraction Consequence given ETC

Here’s the information we can get from analyzing school data…

School #1: Breakdown of Referrals. Total # referrals: 221
School #1: Breakdown of Referrals *Total # referrals: 221 *76 out of 298 students referred at least 1 time 55% of students received 1 referral (42 students) 17% received (13 students) 9% received (7 students) 7% received (5 students) 2.6% received (2 students) 2.6% received (2 students) 1.3% received (1 student) 1.3% received (1 student) 1.3% received (1 student)

School #2: Breakdown of Referrals
Total # referrals: 455 138 out of 283 students received at least 1 referral 40% of students received 1 referral (55 students) 18% received 2 (25 students) 12% received 3 (16 students) 10% received 4 (14 students) 5% received 5 (7 students) 4% received 6 (5 students) 3% received 7 (4 students) 3% received 8 (4 students) .7% received 9 (1 student) .7% received 12 (1 student) .7% received 16 (1 student) .7% received 19 (1 student) .7% received 20 (1 student) .7% received 21 (1 student)

Things that make you go hmmm…
School #1 33% of total referrals by 3 kids 25% of total referrals by repeat offenders (3 or more write ups; 19 students) 25.5% of all students written up at least once School #2 42% of total referrals by 56 kids 21.3% of total referrals by 6 kids 48.8% of our kids written up at least once

School #1

School #2

School #1

Another Example Secondary School Data

Copyright: Kristin N. Johnson-Gros, Ph.D. NCSP
SIS 2007 Copyright: Kristin N. Johnson-Gros, Ph.D. NCSP

Systematic Screening for Behavior Disorders (SSBD) Audience: students in regular elementary grades (K-6) who may be at risk for developing either externalizing or internalizing behavior disorders. Description: a solution to the problems of under-referral of students who may develop behavior disorders by giving regular classroom teachers uniform behavioral standards for use in reducing the idiosyncratic nature of teacher referrals. This mass screening process, which occurs early in a child's school career, is a multiagent, multimethod approach. The screening occurs in three stages: teacher nominations of groups of children whose characteristic behavior patterns most closely resemble profiles of behavior disorders occurring in the school setting and ranking of those students; screening of students in terms of behavioral severity and defining the content of their behavior problems using a series of ratings items and systematic observation of students using a classroom code and a playground code.

Systematic Screening for Behavior Disorders (SSBD) SSBD has been constructed under the following beliefs: teachers in least restrictive environments are more likely to refer pupils who exhibit externalizing behaviors that they perceive as aversive, while under-referring pupils with internalizing disorders; teacher rankings and ratings combined with direct observation is necessary to assess pupil behavior; and academic engaged time and peer-related social behavior are important indicators. Evidence of Effectiveness: In six separate studies, SSBD provided a reliable procedure for systematically screening and identifying elementary school students who demonstrate potential behavior disorders. In six additional studies, SSBD proved to be an accurate procedure that discriminates potential behavior disorder students from nondisordered or non at-risk students within regular classrooms. Requirements: The only major requirement is mastery of the classroom and playground observation codes in stage three of the screening and identification process.

Evaluation of the School-wide & Classroom Environments
School-wide Evaluation Tool (SET) Pbis.org Effective Behavior Support Survey (EBS) Classroom Management Checklist Sprick & Garrison (2008). Interventions: Evidence-based behavioral strategies for individual students. Mississippi Department of Education approved forms

Many Thanks! MDE Staff! All the trainers! Training Materials
Kris Kaase Trecina Green Rhonda Smith All the trainers! Dale Bailey R. Anthony Doggett Brad Dufrene Jennifer Griffin Michael Mong Kristi Mong Ken Swindol Training Materials AIMSweb DIBELS

Questions

Universal Screening At Tier 1
Mississippi Department of Education 2008

Universal Screening At Tier 1

Similar presentations

Presentation on theme: "Universal Screening At Tier 1"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Universal Screening At Tier 1

Similar presentations

Presentation on theme: "Universal Screening At Tier 1"— Presentation transcript:

Similar presentations

About project

Feedback