Comparing two vocabulary tests and TOEIC Masaya Kanzaki Kanda University of International Studies 12:45-1:10, Room MR35

“In English, there is a relatively close relationship between how many words you know, as measured on the standard vocabulary tests, and how well you perform on reading tests, listening tests and other formal tests of your English ability.” Nation & Meara (2002). Vocabulary. In N. Schmitt (Ed.), An introduction to applied linguistics.

Vocabulary Levels Test 1 business 2 clock part of a house 3 horse animal with four legs 4 pencil something used for writing 5 shoe 6 wall

Vocabulary Size Test 1. see: They saw it. a. cut b. waited for c. looked at d. started

Purpose To compare scores on the Vocabulary Levels Test (VLT), the Vocabulary Size Test (VST), and the TOEIC and see how closely they are related

Participants 140 university students who enrolled in 5 sections of a TOEIC preparatory course in April , 2011 International Communication majors 99 second year, 19 third year, and 22 fourth year students 102 female and 38 male

Materials 4 levels of the VLT Version 1 (2000, 3000, 5000, and Academic Vocabulary, 30 questions per level, 120 questions in total) 5 levels of the VST (from 1st to 5th 1000, 10 questions per level, 50 questions in total) TOEIC (100 listening questions: score range 5-495, 100 reading questions: score range 5-495, Total score range 10-990)

Procedures – VLT Done in class on Day 2 (April 13-15) 40 minutes for 4 levels (120 questions) A separate answer sheet was used. An answer key was provided and students scored the test themselves. The answer sheets were collected. 133 students took the test but 6 of them did not complete it. (N = 127)

Procedures – VST Done in class on Day 3 (April 18-20) 25 minutes for 5 levels (50 questions) A separate answer sheet was used. An answer key was provided and students scored the test themselves. The answer sheets were collected. 127 students took the test but 5 of them did not complete it. (N = 122)

Procedures – TOEIC Institutional Program (IP test) on campus on May 27, 2011, 5 p.m.-7:30 p.m. Students had to pay 3,700 yen to take the test. 2 hours, 200 questions The results were provided by the TOEIC organizer, IIBC, two weeks after the test. 62 students took the test. (N = 62)

Results

Descriptive Statistics (N = 127) VLT Descriptive Statistics (N = 127) VLT 2000 VLT 3000 VLT 5000 VLT AV VLT Total Mean 25.00 18.61 16.43 19.28 79.32 Median 25 19 16 20 79 Mode 18 SD 3.46 4.83 4.01 4.54 13.70 Variance 11.95 23.33 16.12 20.57 187.79 Range 22 23 Min 12 7 4 36 Max 30 29 27 115

VLT Histogram (N = 127)

VLT Correlations (N = 127) VLT 2000 VLT 3000 VLT 5000 VLT AV VLT Total .49* .35* .54* .71* .57* .65* .86* .60* .78* .87* * p < .01

Descriptive Statistics (N = 122) VST Descriptive Statistics (N = 122) VST 1000 VST 2000 VST 3000 VST 4000 VST 5000 VST Total Mean 8.99 6.20 6.86 5.54 4.63 32.22 Median 9 6 7 5 32 Mode 4 33 SD 0.90 1.30 1.27 1.23 1.42 3.50 Variance 0.82 1.70 1.63 1.52 2.02 12.22 Range 18 Min 3 1 23 Max 10 8 41

VST Histogram (N = 122)

VST Correlations (N = 122) 1 .18 .07 -.07 .08 .36* .20 .30* .34* .74* VST Total 1 .18 .07 -.07 .08 .36* .20 .30* .34* .74* -.01 .49* .22* .53* .66* * p < .01

Descriptive Statistics (N = 62) TOEIC Descriptive Statistics (N = 62) TOEIC L TOEIC R TOEIC Total Mean 350.24 246.61 596.85 Median 352.5 250 595 Mode 405 255 590 SD 47.70 55.37 92.34 Variance 2275.76 3065.39 8527.24 Range 230 280 495 Min 225 130 370 Max 455 410 865

TOEIC Histogram (N = 62)

TOEIC Correlations (N = 62) TOEIC L TOEIC R TOEIC Total 1 .60* .88* .91* * p < .01

VLT vs. VST Correlations (N = 109) VLT 2000 VLT 3000 VLT 5000 VLT AV VLT Total VST 1000 .11 .33* .38* .34* .37* VST 2000 .35* .28* .27* VST 3000 .06 .13 .22 .16 .18 VST 4000 .24* .21 .30* .32* VST 5000 .31* .17 VST Total .41* .46* .43* .50* .57* * p < .01

VLT vs. VST

VLT vs. TOEIC Correlations (N = 59) TOEIC L TOEIC R TOEIC Total .27 .50* .44* VLT 3000 .31* .74* .61* VLT 5000 .42* .66* .62* VLT AV .47* .69* VLT Total .45* .80* .72* * p < .01

VLT vs. TOEIC

VST vs. TOEIC Correlations (N = 54) TOEIC L TOEIC R TOEIC Total .11 .31 .25 VST 2000 .27 .30 .33* VST 3000 .17 .04 .12 VST 4000 .15 .29 .26 VST 5000 .28 .36* VST Total .38* .48* .49* * p < .01

VST vs. TOEIC

VLT & VST vs. TOEIC Correlations (N = 51) 1 .52* .98* .36* .79* .68* VLT Total VST Total VLT&VST TOEIC L TOEIC R TOEIC LR 1 .52* .98* .36* .79* .68* .67* .32 .48* .46* .38* .69* .85* .89* * p < .01

VLT & VST vs. TOEIC

Why does the VLT correlate better with the TOEIC? The number of questions Test-toughness Anything else?

References Beglar, D. (2010). A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27 (1), 101–118. Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the vocabulary levels test. Language Testing, 18 (1), 55–88.