An Investigation of Cognitive Operations on L2 Listening Comprehension Performance Speaker: Dr. Hui-Fang Shang ( 尚惠芳博士 ) Outline: I.Introduction II.Literature Review III.Purpose of Study IV.Methodology V.Results VI.Discussion and Implications

1 I. Introduction * Hard to evaluate EFL learners’ listening comprehension performance due to lack of a valid listening test * To develop a valid listening comprehension test Need to understand the listening processes and how listening functions * Hard to evaluate EFL learners’ listening comprehension performance due to lack of a valid listening test * To develop a valid listening comprehension test Need to understand the listening processes and how listening functions

II. Literature Review * Listening comprehension processes: -- Retain information in the memory: Deposit raw speech in short-term memory; predict information; recall background information to interpret message; hold information in long- term memory (Duzer, 1997) -- Interact between background knowledge and text: Bottom- up and top-down processing: (Duzer, 1997) -- Use local (links between clauses & sentences) and global (overall coherence) strategies (Van Dijk & Kintsch, 1983) -- Use trivial (numerical details), local, and global strategies (Shohamy, 1991) 2

3 * How L2 listeners deal with text processing: -- Efficient listeners use background knowledge; low listeners use mostly local details (Hildyard & Olson, 1982) -- Efficient L2 listeners activate more L1 knowledge in the form of concept-driven schemata (Wolff, 1987) -- Effective L2 listeners use both top-down and bottom-up strategies; ineffective listeners determine the meanings of individual words (O’Malley, Chamot, & Kupper, 1989) * How L2 listeners deal with text processing: -- Efficient listeners use background knowledge; low listeners use mostly local details (Hildyard & Olson, 1982) -- Efficient L2 listeners activate more L1 knowledge in the form of concept-driven schemata (Wolff, 1987) -- Effective L2 listeners use both top-down and bottom-up strategies; ineffective listeners determine the meanings of individual words (O’Malley, Chamot, & Kupper, 1989) II. Literature Review (cont.)

4 -- Low listeners perform better on local items than on global items (Shohamy, 1991) -- Low listeners do not use appropriate content schemata (Jensen & Hansen, 1995) -- 6 listening functions to comprehend a message: Identification, orientation, main idea comprehension, detail comprehension, full comprehension, and replication (Lund’s Taxonomy, 1990) -- Low listeners perform better on local items than on global items (Shohamy, 1991) -- Low listeners do not use appropriate content schemata (Jensen & Hansen, 1995) -- 6 listening functions to comprehend a message: Identification, orientation, main idea comprehension, detail comprehension, full comprehension, and replication (Lund’s Taxonomy, 1990)

-- 4 similar categories to comprehend a text : (Dunkel, Henning, & Chaudron, 1993) ◎ Orientation meanings: Understand persons & their relationships ◎ Detail meanings: Simple lexical meanings ◎ Main ideas: Major points ◎ Implications: Applications of background knowledge -- Those 4 categories of understanding the text meaning: “Cognitive tasks” -- The ability to “identify” and “interpret” the cognitive tasks: “Cognitive operations” II. Literature Review (cont.) 5

6 III. Purpose of Study * Use Dunkel, Henning, & Chaudron’s (1993) theory of cognitive operations to measure EFL listeners’ comprehension competence * Select 3 components of cognitive operations: (1) Interpreting main ideas (local) (2) Identifying details (trivial) (3) Interpreting implications (global) * Use Dunkel, Henning, & Chaudron’s (1993) theory of cognitive operations to measure EFL listeners’ comprehension competence * Select 3 components of cognitive operations: (1) Interpreting main ideas (local) (2) Identifying details (trivial) (3) Interpreting implications (global)

III. Purpose of Study (cont.) * Explore 3 main research questions: (1) What is the most difficult cognitive operation in terms of those 3 components of cognitive operations? (2) For different proficiency listeners, what is their listening performance? (3) How are listeners’ perceptions regarding their listening performance consistent with the quantitative research results? 7

IV. Methodology 8 * Subjects: -- Include 63 sophomores of Applied English Department at I-Shou University -- Divide subjects into 3 proficiency groups: Low, intermediate, high groups based on the scores of simulated TOEFL listening test (from 6 to 13 out of 13, Mean=10.48, Median=10) * Subjects: -- Include 63 sophomores of Applied English Department at I-Shou University -- Divide subjects into 3 proficiency groups: Low, intermediate, high groups based on the scores of simulated TOEFL listening test (from 6 to 13 out of 13, Mean=10.48, Median=10) Table 1 Number of Subjects and Scores of Each Proficiency Group LowIntermediateHigh Scores6~91011~13 Number (% of total sample) 13 (21%) 20 (32%) 30 (47%) Total63

IV. Methodology (cont.) * Materials: -- Select 3 extended conversations with the similar topic of campus events from the textbook of “The Heinle & Heinle TOEFL Test Assistant: Listening (Broukal, 1995) -- Choose 3 listening texts containing: ◎ Paraphrasing : Understand main ideas (local strategy) ◎ Listening for details: Answer detail information (trivial strategy) ◎ Making inferences: Draw conclusions (global strategy) 9

10 * Procedures: Construct 12 items in total and listen to the texts twice * Measurement Instruments & data analyses: -- Use 2 scoring methods: binary (correct/incorrect) of item scores and rating scale (a self-perceived survey) -- Investigate mean differences among 3 cognitive operations on 3 proficiency groups by computing a one-way ANOVA analysis -- Examine which cognitive operation yields higher listening performance by employing a Post Hoc test -- Administer a self-rating questionnaire to probe subjects’ perceptions of the difficulty level of each cognitive operation * Procedures: Construct 12 items in total and listen to the texts twice * Measurement Instruments & data analyses: -- Use 2 scoring methods: binary (correct/incorrect) of item scores and rating scale (a self-perceived survey) -- Investigate mean differences among 3 cognitive operations on 3 proficiency groups by computing a one-way ANOVA analysis -- Examine which cognitive operation yields higher listening performance by employing a Post Hoc test -- Administer a self-rating questionnaire to probe subjects’ perceptions of the difficulty level of each cognitive operation IV. Methodology (cont.)

11 IV. Methodology (cont.) * 5 hypotheses: (1) Interpreting implications is the most difficult question type. (2) High proficiency listeners can successfully use both bottom-up and top-down processing to determine major, literal, and implied meanings than the other two groups. (3) The intermediate listeners’ performance regarding those three cognitive operations will be better than the low listeners’. (4) The low proficiency listeners will perform better on items referring to detail questions than on items referring to main idea and inference questions. (5) Listeners’ perceptions will be consistent with the experimental research results. * 5 hypotheses: (1) Interpreting implications is the most difficult question type. (2) High proficiency listeners can successfully use both bottom-up and top-down processing to determine major, literal, and implied meanings than the other two groups. (3) The intermediate listeners’ performance regarding those three cognitive operations will be better than the low listeners’. (4) The low proficiency listeners will perform better on items referring to detail questions than on items referring to main idea and inference questions. (5) Listeners’ perceptions will be consistent with the experimental research results.

12 Table 2 Classification of Scores Obtained according to Three Question Types for Three Proficiency Groups Table 2 Classification of Scores Obtained according to Three Question Types for Three Proficiency Groups V. Results Low Mean (%) Intermediate Mean (%) High Mean (%) Main idea (local) Detail (trivial) Inference (global) * Each group performs worst on the detail questions ~ Does not support Hypotheses 1 & 4 * High group performs best than the other two groups ~ Support Hypothesis 2 * Intermediate group performs worse than low group except main idea questions ~ Broadly not support Hypothesis 3

13 Table 3 A T-Test Analysis between Local (Main Idea) and Global (Inference) Questions among Three Groups V. Results (cont.) MeanNSD SE Mean tdfSig. Low Main idea Inference * Intermediate Main idea Inference * High Main idea Inference * * The means for the main idea questions are higher than the inference ones except for the low group ~ broadly support Hypothesis 1

14 V. Results (cont.) Table 4 A One-way ANOVA Analysis of Those Three Proficiency Groups Table 4 A One-way ANOVA Analysis of Those Three Proficiency Groups ScoreSSdfMSFSig. Between Groups Within Groups Total * There is a significant difference at the.001 level for those three cognitive operations on those three proficiency groups.

15 Table 5a A Post Hoc Test among Low, Intermediate, and High Groups in Three Cognitive Tasks V. Results (cont.) GroupNMeanSD Main Idea Low Intermediate High Detail Low Intermediate High Inference Low Intermediate High * The high group outperforms the intermediate and low groups in those 3 cognitive tasks ~ Support Hypothesis 2 * The low group outperforms the intermediate one in detail and reference sections ~ Broadly not support Hypothesis 3

16 Table 6a Self-Report of Difficulty Level for Main Idea Questions V. Results (cont.) Very easy EasyOKDifficultVery difficult Low008 (66.7%)3 (25%)1 (8.3%) Intermediate0016 (84.2%)3 (15.8%)0 High2 (4%) 36 (72%)10 (20%)0 * The low group considers the main idea questions as the most difficult ones, followed by the high group, followed by the intermediate group ~ Partially consistent with Table 5a (mean scores: low ＜ intermediate ＜ high)

17 Table 6b Self-Report of Difficulty Level for Detail Questions V. Results (cont.) Very easy EasyOKDifficultVery difficult Low007 (53.8%)5 (38.5%)1 (7.7%) Intermediate0012 (66.7%)6 (33.3%)0 High01 (2%)31 (63.3%)17 (34.7%)0 * The low group considers the detail questions as the most difficult ones, followed by the high group, followed by the intermediate group ~ Not consistent with Table 5a (mean scores: intermediate ＜ low ＜ high)

18 Table 6c Self-Report of Difficulty Level for Inference Questions V. Results (cont.) Very easy EasyOKDifficultVery difficult Low02 (15.4%)6 (46.2%)4 (30.8%)1 (7.7%) Intermediate0016 (84.2%)3 (15.8%)0 High1 (2%)3 (6.1%)32 (65.3%)13 (26.5%)0 * The low group considers the inference questions as the most difficult ones, followed by the high group, followed by the intermediate group ~ Not consistent with Table 5a (mean scores: intermediate ＜ low ＜ high)

19 Table 7a Frequency of Three Question Types for Low Proficiency Group V. Results (cont.) Very easy EasyOKDifficultVery difficult Main idea008 (66.7%)3 (25%)1 (8.3%) Detail007 (53.8%)5 (38.5%)1 (7.7%) Inference007 (53.8%)5 (38.5%)1 (7.7%) * The main idea questions are the least difficult ones. ~ Not consistent with Table 2 (The inference questions are the least difficult ones.)

20 Table 7b Frequency of Three Question Types for Intermediate Proficiency Group V. Results (cont.) Very easy EasyOKDifficultVery difficult Main idea0016 (84.2%)3 (15.8%)0 Detail0012 (66.7%)6 (33.3%)0 Inference0012 (66.7%)6 (33.3%)0 * The detail and inference questions are (very) difficult for intermediate group. ~ Partly supports Hypothesis 1 that interpreting implications is the most difficult question type.

21 Table 7c Frequency of Three Question Types for High Proficiency Group V. Results (cont.) Very easy EasyOKDifficultVery difficult Main idea2 (4%) 36 (72%)10 (20%)0 Detail01 (2%)31 (63.3%)17 (34.7%)0 Inference01 (2%)31 (63.3%)17 (34.7%)0 * The detail and inference questions are (very) difficult for high group. ~ Partly supports Hypothesis 1 that interpreting implications is the most difficult question type.

22 VI. Discussion and Implications 1.Each group performs worst on the detail questions (see Table 2) because it’s difficult to memorize all the information from the listening texts. => Irrelevant recall of names of numerical data may distract listeners’ attention on the whole comprehension. => Since such cognitive operation serves no meaningful purpose as evaluating listening comprehension, it’s recommended to avoid the insignificant numerical details on listening comprehension tests.

23 VI. Discussion and Implications (cont.) 2. Except for low group, subjects perform better on local (main idea) questions than on global (inference) ones. => Comprehension of local information is more attainable than that of macro information (Shohamy, 1991). => It’s more difficult to employ the concept to generalize, infer, and synthesize the information from the cognitive processing than to get the main idea information only (Van Dijk & Kintsch, 1983).

24 VI. Discussion and Implications (cont.) 3. There is an inconsistency for the low group regarding the performance of global and local questions from the quantitative and qualitative research results. => The low listeners may guess the answers mostly since it’s a multiple-choice test. => It’s appropriate to use more test types to certainly reflect listening comprehension competence.

25 VI. Discussion and Implications (cont.) 4. The low group outperforms the intermediate group in the detail and inference sections. => The listening texts which were used to affect the degree of listening comprehension are relatively difficult to the immediate group. => Other factors, such as familiarity with the topic, background knowledge, text density, etc. may affect listeners’ performance. => Future research should look at the influence of those factors on the listening comprehension performance.

26 VI. Discussion and Implications (cont.) 5. To result in more construct valid listening comprehension tests, teachers should carefully select a variety of testing instrument with various test types and question types, to better reflect the trait of L2 listening comprehension.