2.E Developing an Assessment Framework

Starting from Strand A: Modules Cognitive instruments developed under Strand A for 15-year-old students attending educational institutions in grades 7 and higher that are comparable to main PISA. Cognitive instruments developed under Strand A for 15 year olds that are in grades 6 or 5 and even lower grades that are part of a national component. Cognitive instruments (reading components) developed specifically under Strand C that are targeted on the out-of-school 15 year olds who have never been to school or who have dropped out after completing only lower grades.

Types of Cognitive Instrument Specific to Strand C Need a range of assessment instruments specifically for Strand C : Basic assessment based on PASEC / SACMEQ items Basic literacy and numeracy assessment such as ASER-type or EGRA early years assessment Pictorial / verbal assessment A set of channels/ filters /gateways will be needed to identify appropriate instrument

Assessment Experience ASER and UWEZO are basic surveys of child literacy and numeracy, most likely to provide basic material for those who have never been to school or dropped out after a few grades PASEC and SACMEQ used in SSA are school based; their material likely to be useful for group that has completed primary (Strand A) LAMP, PIAAC and STEP are oriented towards adults, but there are some elements which should be considered by a contractor. In particular, both LAMP and PIAAC have items on Reading components

Other Assessment Issues None of the other surveys reviewed have paid specific attention to the plight and treatment of children with special needs (e.g. disability). This pilot exercise is seen as an opportunity to partially remedy those omissions. The languages and/or culture of the parent/ caregiver generation may be very different from that of the 15 year old

ASERUWEZOSACMEQ Levels and Categorisation Levels Letters: Set of commonly used letters.Recognise Letters1: Pre‐reading ■ Words: Common familiar words with 2 letters and 1 or 2 matras. Read selected words 2: Emergent Reading ■ Level 1 (Std 1) text: Set of 4 simple linked sentences, each having no more than 4-5 words*. Read one of two paragraphs3: Basic Reading ■ Level 2 (Std 2) text: Short story with 7-10 sentences. Sentence construction is straightforward, words are common and the context is familiar to children*. Read a story and answer 2 comprehension questions 4: Reading for meaning Procedure for CategorisationCategorisation 5: Interpretive reading Children are offered a paragraph to read. If they can do either of those easily, they are then offered a story to read; and if they can do that easily they are rated at Story level; if not they are rated at Paragraph level. If they cannot read the Paragraph easily, then they are offered a word list. If they can answer 4 out of 5 items but not read the paragraph, they are rated at Word level. If they cannot read 4 out of the 5 chosen words then they are presented with a list of letters, from which they choose 5 items. If they can answer 4 out of 5 chosen items they are rated as at Letter level if they cannot, they are rated as at Beginner level. Children are categorized according to the highest level attained. For example, if a child could read the words but not the paragraph, that child was ranked at word level 6: Inferential Reading Pass Levels 7: Analytical Reading A child “passed” the literacy test if he/she was able to read the story aloud and correctly answer the two comprehension questions 8: Critical Reading

Reading Components in LAMP 1.Vocabulary 2.Digit naming 3.Rapid digit naming 4.Lowercase letter naming 5.Uppercase letter naming 6.Rapid letter naming (lowercase) 7.Word recognition (high frequency words; lowercase) 8.Rapid word recognition (same as previous) 9.Decoding (rare words; non-words) 10.Rapid decoding (same as previous) 11.Sentence processing (oral reading; silent reading, timed) 12.Passage fluency (oral reading; silent reading, timed)

Language Issues Language of instruction for in-school 15 yos inappropriate medium for testing some of target group in Strand C. Here, we distinguish four sub- divisions of the target group: 1.First, a small group which will have had some secondary and then dropped out. 2. Second, a minority will have completed primary and then dropped out. 3.Third, a significant minority will have dropped out before completing primary 4.Fourth, a small minority (in most countries) will have never been to school

Language Issues (continued) Language of instruction appropriate for first two groups, except where entire primary cycle conducted in local language (mother tongue), possibly for third group, but clearly not for 4 th Generally agreed that four grades sufficient for learning language Breakdowns of those currently OOS suggest that very high percentages (70% to 90%) are less than 5 th grade

Testing Procedures/ Filters Age estimation: include specific event calendar. To avoid over-identification of 15 yos, calculate expected number of ‘hits’ in each area and explain to citizen volunteers Status vis-à-vis school: in addition to direct questions to 15 yo and caregiver, need to ask about any other learning experiences, reasons why this 15yo no longer in school and about education/ schooling of other children in household

Structure of tests For Language EITHER directly ask 15yo which language they are comfortable with OR use ASER/ UWEZO approach to identify level of literacy (letters, words, sentences) For mathematics, OOS 15 yo will almost certainly understand verbally numbers because of financial transactions, but may not be able to read them. Use ASER/ UWEZO approach to identify level of numeracy. If 15yo unable to answer simplest item, possibility of questions based on graphic representations and non-verbal reasoning; both would have to be made equivalent

Questions for Discussion Starting from Strand A: Is it just literacy and numeracy? What do we additionally learn from other assessments? What can be achieved in minutes? How can Strand C handle multiple languages? What do we call Strand C? Pre-PISA? How do we adapt to 15 year olds with special needs? How do we adapt to different cultural contexts in the households?