1 全国高等学校英语教师 教育与发展系列研修班 Statistics in action: A listening strategies project 25 July 2010 Victoria University of Wellington 中国外语教育研究中心 顾永琦

In this session…  Part 1: Feedback to your journals  Part 2: Exercises 1 and 2 demo  Part 3: Statistics in a listening strategy project

Part 1 Your journals  收获  问题  建议

问题  运用实例，看什么研究问题用什么统计方 法，生成什么样的图表，如何解读  一些关键术语还是模糊，如 nominal, ordinal, interval scales, p 值，相关正负值  Comment ： realistic expectations

建议  加强练习与实际操作  SPSS 操练手册？  提供 references  提前把材料放在网上

Part 2 Exercises 1 and 2 demo

 If you run Frequencies on the data, you get the descriptive statistics for all 44 cases, i.e., for both classes.  What you need is mean scores and SD for both classes.  You can get the descriptive statistics you want from the Explore command under Descriptive statistics.

Reporting your descriptive statistics ClassN 均值标准差 Professor's Class Graduate Student's Class

Interpreting your findings  The mean score of the graduate student’s class (Mean=58.13) was indeed higher than that of the professor’s class (Mean=55.29). In general, the graduate student’s class outperformed the professor’s class.  The professor’s class also had a much larger standard deviation than the student’s class, indicating that the professor’s class showed a wider spread of scores than the other class.  NB: you can also show the spread visually on a histogram.

Exercises 2  Correlation

Choose your options to include descriptive statistics

Report the descriptive statistics first

Report the correlation table

Interpreting your correlation findings  The findings showed that candidates’ abilities were positively correlated with the salary package offered to them. All four ability measures were found to be significantly correlated to the candidates’ annual salary.  The highest correlation existed between the candidates’ IQ scores and their salary (r=.844, p<.001). In other words, someone’s IQ score explains about 65% of the salary s/he earns. Communication score revealed the second highest correlation with salary (r=.813, p<.001)…  This is encouraging news for business school students, in that hard work, good marks, and an intelligent mind are indeed highly related to the salary they get after they graduate.

23 Part 3: Stats in action 3.1. Developing and validating the Strategy Battery for Young Learners of English (SBYLE) 3.2. Statistics in a listening strategy project

24 Strategy Battery for Young Learners of English (SBYLE) Purpose: Profiling English language learner strategies used by upper primary school pupils in Singapore 1. Listening Strategy Questionnaire 2. Reading Strategy Questionnaire 3. Writing Strategy Questionnaire

25 Designing the SBYLE  The listening strategies questionnaire as an example

26 Version numbers for the Strategy Battery for Young Learners of English VersionPurpose 1.0Creation of questionnaire items 1.1Validation by experts 1.2Validation by school teachers 1.3Validation by target sample (N=6) 1.4Piloting among large sample (N=293) Ensuring internal consistency reliability 2.0Final administration (N=3,618)

27 Version 1.0: Creation of questionnaire items What was doneCriteria 1.Top-down: metacognitive, cognitive, and social/affective 2.Bottom-up: Turn Phase I codings into questionnaire items. 3.List items in categories 4.Find items in existing literature, integrate them into what has been created, under existing or new categories. 1.Minimum 3, maximum 10 in each category, no single item variables. (strategies are latent variables: A strategy is not a single instance of behaviour) 2.Closed, 5-point Likert scale: Never to Always 3.Include a purpose, if possible, in each item. (strategies are purposeful)

28 Where do Questionnaire items come from? Phase I studies  Participants: 4 schools, all 6 grades in each school, one high, one medium, and one low EL proficiency pupil from each grade (4x6x3=72)  Materials:  Listening, reading, writing tasks  Think-aloud protocols  Analysis:  Coding and pattern exploration

29 Version 1.0 Example Strategy typeStrategiesItems Metacognitive Self- initiation 1.I look for opportunities to listen in English. 2.To improve my listening in English, I watch English TV programmes. 3.When I’m free, I find interesting things to listen to in English (for example, TV, radio, etc). 4.I try to find out how to improve my listening in English.

30 Validating the SBYLE

31 Validity checks  Content validity Consult experts, teachers, target group on 1) relevance 2) coverage 3) representativeness, and 4) exactness of wording  Construct validity  Comparison with theory  Factor analysis  Response validity  Participating pretest: paired think-aloud

32 Reliability checks Type of reliabilityDecision Internal consistency reliability yes Test-retest reliability no

33 Version 1.1: Validation by experts What was doneGuiding questions Item-by-item examination and change (What do we intend to cover?) Conceptual coverage: Do items under each category belong there? Should there be other items? Should there be other categories? Wording: Is wording for each item clear? Is wording for each item simple enough to be understood by upper primary school pupils?

34 Changes made at this stage: Examples OriginalRevised When I listen to something, I try hard to understand every word before I can put these words together to understand the whole thing. When I listen, I pay attention to every word. When I listen, I try to form images in my mind for better understanding. When I listen, I try to form pictures in my mind in order to understand better. I try to relate what I hear to my own experiences in order to learn something from what I hear. I try to see connections between what I hear and my experiences.

35 Version 1.2: Validation by teachers What was doneGuiding questions Consultation with teachers (intended meaning = perceived meaning?) 1.Is wording for each item clear? 2.Is wording for each item simple enough to be understood by upper primary school pupils? 3.If the wording of an item is not clear enough, what alternative suggestions would you make?

36 Version 1.3: Validation by target sample (intended meaning = perceived meaning?) What was doneGuiding questions 1.Randomize items in each questionnaire 2.The formal version for small-scale piloting among 6 pupils item by item 3.Design answer sheets 1.Do pupils understand each item? 2.Are there items that are easily misunderstood? 3.Do pupils have problems with the answer sheet? 4.Are there any other snags? 5.How much time is needed to answer each questionnaire?

37 Version 1.4: Construct validity and internal consistency reliability  Formal piloting among 293 pupils (P4-P6)  Elicit student and teacher comments  Revise, after factor analysis and item analysis: delete, add, change, collapse categories, re-categorize items  Fine-tune survey administration procedures and detect problems (Guidelines for Questionnaire Administration)Guidelines for Questionnaire Administration

38 Version 1.4: Reliability analysis VariableItem # in pilot Q’naire Item in original categories Item-Total correlation Alpha if item deleted Scale alpha Inferencing2 delete When I don’t understand something, I use my knowledge about the English language to guess When I don’t understand something, I use what I have already heard to guess When I don’t understand something, I make several guesses When I don’t understand something, I use my general knowledge to make a guess delete When I don’t understand something, I use my knowledge about text structure to help me understand When I don’t understand something, I use my knowledge about the topic to guess

39 Deleted item: #38 Item in original categories Item-Total correlation Alpha if item deleted When I don’t understand something, I use my knowledge about text structure to help me understand

40 Deleted item #2 Item in original categories Item-Total correlation Alpha if item deleted When I don’t understand something, I use my knowledge about the English language to guess

Obtaining variables from individual items  COMPUTE LSELFINI=(L11+L12+L14 +L18)/4.  COMPUTE LPLAN=(L3+L15+L23)/3.  COMPUTE LMONITOR =(L21+L32+L5+L9+L36)/5.  COMPUTE LPERCEPT =(L8+L24 +L34+L37)/4.  COMPUTE LINFER =(L1+L4+L17+L26 +L31+L27+L33)/7.  COMPUTE LPREDICT =(L6+L7+L13+L29+L35)/5.  COMPUTE LUTILISE =(L16+L19+L20+L22+L30)/5.  COMPUTE LSOCIAL=(L2+L10+L28+L25+L38)/5.

Running item analysis and reliability in SPSS  What we are doing: 1. Checking if items that make up a variable are consistent among themselves. 2. Doing item-total correlation to see how each item in the variable is correlated to the total variable score. 3. Determining which item should be removed from the variable (alpha is item deleted)  Example: the monitoring variable LMONITOR was made up of 5 items L21, L32, L5, L9, and L36.

Getting internal-consistency reliability

Locate all items in the variable

Specifying the statistics you want

Deciding if any item does not belong to the variable

48 Version 1.4: Problems and solutions Administration problems (N=293) Solutions 1.Pupils don’t remember ID numbers 2.Wrong understanding of “Grade” 3.Wrong answer sheets used 1.Ask schools to remind students in advance 2.Provide P4/P5/P6 choices 3.Tell Teachers in advance that different questionnaires require different answer sheets

49 Version 2.0: Final Administration Number of Pupils Percent School School School School School School Total

Statistics in the listening strategy project: Deciding what stats to use Research questionStatistics 1. What is the listening strategy use pattern among Singapore upper primary school pupils? Descriptive statistics 2. How are listening strategies related to English language results? Correlation, Regression (strength of associations between the two vars) or ANOVA (locating differences between top, middle, & bottom groups) 3. Do strategies change from grade to grade? ANOVA (differences among three grades)

Reporting descriptive statistics: The table format

Interpretation and textual report

Listening strategies and English grades

Textual interpretation

Run your own item analysis and Chronbach’s alpha in SPSS  Items 15, 17, 14, 25, 5, 8, 11, 34, 26  Items 1, 2, 3, 6, 18  Items 24, 29, 16, 19  Items 28, 27, 21  Items 30, 31, 32  Items 7, 10, 4, 33, 20  Items 23, 12, 22