1. Issues in Assessing Language Proficiency and Academic Achievement Lyle F. Bachman Department of Applied Linguistics & TESL University of California, Los Angeles

2. Overview F ESEA mandate F Nature of language use and language ability F Issues in assessing academic English proficiency F Measuring English proficiency as opposed to academic achievement F Current assessments of English proficiency for ELLs F Conclusion

3. ESEA Mandate F Assess “academic English proficiency” Oral languageOral language ReadingReading Writing skillsWriting skills F Assess at all levels, from beginning to advanced, K-12 F Assess annually F Assessments must be sensitive to annual increases in proficiency

4. Nature of Language Use F Language is used to accomplish things in the real world. F Language use is the situated negotiation of meaning through language. F Language use involves: Performing functions that arePerforming functions that are Realized as styles or registers, andRealized as styles or registers, and Expressed through a formal code.Expressed through a formal code.

5. Language Functions F Language functions are the social uses to which we put language. (Halliday 1976). F Four macro-functions (Bachman, 1990) Ideational Ideational Manipulative Manipulative Heuristic Heuristic Imaginative Imaginative F General classroom functions (Valdez Pierce and O’Malley, 1992) F Scientific language functions (Chamot & O’Malley, 1987)

6. Register and Style F Register: a variety of language that is associated with a particular context or domain. (Halliday et. al. 1964) Distinguished essentially by grammar and vocabularyDistinguished essentially by grammar and vocabulary F Style: a variety of language used by individuals, appropriate to levels of formality. Ranges from intimate to consultative to oratorical (Joos 1967)Ranges from intimate to consultative to oratorical (Joos 1967)

7. Formal Code F Formal code: the system of language that we use to express functions in specific contexts. Grammatical structuresGrammatical structures VocabularyVocabulary  Technical  Subtechnical  General, with field specific meaning  General Cohesive markersCohesive markers Rhetorical organization/conversational structureRhetorical organization/conversational structure

8. Language ability (proficiency) F Knowledge of functions, register, formal code F Capacity for implementing this knowledge in language use

9. Issues in assessing academic English proficiency 1. Defining the construct, “academic English proficiency” 2. Specifying assessment tasks 3. Setting performance standards

10. Defining the construct, “academic English proficiency” Several different approaches to this 1. Academic language as a compilation of unique language functions and structures that are difficult for language minority students to master (Hamayan and Perlman, 1990)

11. Defining the construct, “academic English proficiency” 2. Academic language as distinct from interpersonal conversational language (Cummins, 1980, 1983; Mohan, 1986) Amount of cognitive demandAmount of cognitive demand Degree of contextualizationDegree of contextualization Basic interpersonal communication skills (BICS): cognitively undemanding and context-embeddedBasic interpersonal communication skills (BICS): cognitively undemanding and context-embedded Cognitive academic language proficiency (CALP): cognitively demanding and context-reducedCognitive academic language proficiency (CALP): cognitively demanding and context-reduced

12. Defining the construct, “academic English proficiency” 3. Academic language as language in interaction (e.g., Mehan, 1979) Turn-takingTurn-taking Patterns of participationPatterns of participation Initiation-Reply-Evaluation sequence characterizes classroom discourseInitiation-Reply-Evaluation sequence characterizes classroom discourse

13. Defining the construct, “academic English proficiency” 4. Academic language as stylistic registers (Solomon & Rhodes, 1995) Styles tied to specific academic tasksStyles tied to specific academic tasks Stylistic registers associated with broad, discourse levels of language; not limited to sentence-level linguistic featuresStylistic registers associated with broad, discourse levels of language; not limited to sentence-level linguistic features

14. Specifying assessment tasks 1. “Skills” versus “activities” Another way to conceptualize speaking, reading, writing: as activities we perform with languageAnother way to conceptualize speaking, reading, writing: as activities we perform with language Most theoretical models of speaking, reading and writing are models of processes, rather than of skill or ability.Most theoretical models of speaking, reading and writing are models of processes, rather than of skill or ability.

15. Specifying assessment tasks 2. Language use task “an activity that involves people in using language for the purpose of achieving a particular goal in a particular situation” (Bachman & Palmer, 1996) 3. Use framework of task characteristics to describe language assessment tasks as specific instances of language use tasks

16. Language proficiency and assessment tasks The way we define the construct, “academic English proficiency” and the way we specify assessment tasks will determine the kinds of inferences we can make, and the domains to which these inferences generalize.

17. Setting performance standards 1. ELLs vs. “native English speakers” Different standards?Different standards? Continuous scale of ability?Continuous scale of ability? 2. Performance standard “Educated native speaker”?“Educated native speaker”? Based on professional content standards (e.g., TESOL, 1997) or state standards?Based on professional content standards (e.g., TESOL, 1997) or state standards?

18. Measuring English proficiency as opposed to academic achievement Two-part conundrum: 1. Distinguishing English proficiency from content knowledge 2. Designing assessment tasks that are appropriate for the construct definition

19. Measuring English proficiency as opposed to academic achievement Distinguishing English proficiency from content knowledge: three options (Bachman & Palmer, 1996, Bachman, in press) 1. Define construct as English proficiency only (i.e., formal code, functions, register) 2. Define construct as content knowledge only (e.g., knowledge of mathematics, history, science) 3. Define construct as combination: “ability to use English to process information about academic content”

20. Measuring English proficiency as opposed to academic achievement Designing assessment tasks that are appropriate for the construct definition. Two Hypotheses: F The more we design tasks to isolate English proficiency, the more our assessment tasks look like non-language, and the more we limit our inferences to the formal language code, i.e., grammar and vocabulary F The more we try to design tasks to assess aspects of language use (functions, cohesion, rhetorical /conversational organization), the more topical content we include in our tasks

21. Measuring English proficiency as opposed to academic achievement Designing assessment tasks that are appropriate for the construct definition. Grammar Markers of Cohesion Vocabulary Rhetorical Organization Language Functions Register Aspects of English Proficiency Assessed Involvement of Content Knowledge/Topical Content MoreLess

22. Current assessments of English proficiency for ELLs 1. Great deal of variety in areas of language ability and assessment tasks included 2. Tasks generally reflect the hypotheses above 3. Typically difficult to distinguish English proficiency assessment tasks from academic achievement assessment tasks, especially in the area of reading

23. Conclusion 1. Current approaches to assessing English proficiency for ELLs have, in my view, failed to build an adequate validation argument to provide evidence supporting score-based inferences of English proficiency only 2. Assessments of English proficiency based on current approaches (“first generation” assessments) are not likely to resolve this issue.

24. Conclusion 3. For “next generation” assessments, need to conduct research that would investigate: The differential effects of English proficiency and content knowledge on test performanceThe differential effects of English proficiency and content knowledge on test performance The differential effects of a wide range of assessment task types on test performanceThe differential effects of a wide range of assessment task types on test performance Utilize recent developments in measurement theory and test design (e.g., Mislevy et. al. 2002a, 2002b) to disentangle student abilities from task characteristicsUtilize recent developments in measurement theory and test design (e.g., Mislevy et. al. 2002a, 2002b) to disentangle student abilities from task characteristics

25. Grammar Markers of Cohesion Vocabulary Rhetorical Organization Language Functions Register Aspects of English Proficiency Assessed Involvement of Content Knowledge/Topical Content MoreLess