Research Methods of Applied Linguistics and Statistics By Qin Xiaoqing (Huazhong University of Science and Technology)

2 Lecturer: 秦晓晴 Public Password:

3 An Introduction to this Course Definitions & classifications of research, and research procedures The concept of variables Quantitative research design Questionnaire survey Item analysis,validity & reliability Data exploring

4 Descriptive statistics Inferential statistics Writing a research proposal Reporting a research Research evaluation

5 Nature of This Course To give lectures To hold discussions and presentations To do self-study To administer a test

6 Course books and references (1) Mackey, A., & Gass, S. M. (2005). Second Language Research : Methodology and Design. Mahwah, NJ: Lawrence Erlbaum. Brown, J. D. (2001). Using Surveys in Language Programs. Cambridge: Cambridge University Press. Dörnyei, Z. (2003). Questionnaires in Second Language Research: Construction, Administration and Processing. Mahwah, N.J.: Lawrence Erlbaum Associates.

7 Course books and references (2) Hatch, E. M., & Lazaraton, A. (1991). The Research Manual: Design and Statistics for Applied Linguistics. New York, NY: Newbury House Publishers. Seliger, H. W. & E. Shohamy, Second Language Research Methods. Shanghai: 上海外 语教育出版社. 刘润清， 1999 ，《外语教学中的科研方法》。北 京：外语教学与研究出版社。 秦晓晴， 2003 ，《外语教学研究中的定量数据分 析》。武汉：华中科技大学出版社。 文秋芳， 2004, 《应用语言学研究方法与论文写 作》。北京：外语教学与研究出版社。

8 Definition of Research 1.Dictionary definitions (1) According to Longman Dictionary, research means advanced study of a subject, so as to learn new facts on scientific laws. (2) Collin’s Dictionary says research is a detailed study of a subject or an aspect of a subject.

9 2. Formal definition of research Research is a systematic approach to finding answers to questions. Research is a systematic approach to finding answers to questions.question—appropriateapproach—systematicanswer—valid

10 Purposes of research (a) increasing knowledge within the discipline of applied linguistics, and (b) increasing knowledge within oneself to understand new developments within the discipline.

11 How to increase knowledge supports the theoretical basis of the discipline providing evidence for the efficacy of a curriculum, EFL teaching and learning, and so on. Developing Research Tools. For most students, the ability to understand research in applied linguistics may be more important than making a research contribution to the profession.

12 Definition of theory Dictionary: A theory is a formal idea or set of ideas that is intended to explain something. Kerlinger (1986: 9): A theory "is a set of interrelated constructs (concepts), definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting the phenomena".

13 Types of research Adapted from Brown 2001:1

14 Characteristics of statstical research Systematic (systematically structured with definite procedural rules) Logical (based on a step-by-step logical pattern) Tangible (based tangible, quantifiable information, ie data) Replicable (able to repeat the research) Reductive (help form patterns in the seeming confusion of facts) (J. D. Brown 1988)

15 Steps of quantitative research (1) Ask or form appropriate questions (2) Review the literature (3) State hypothesis null (H 0 ) alternative (H 1 ): positive, negative null (H 0 ) alternative (H 1 ): positive, negative (4) Select an approach and do the research (5) Analyze the data (6) Report results

16 Sources of Research Questions Step 1: Formulating a general research question Interest and personal experience Journals of our field Supervising faculty Sources outside second language

17 Step 2: Narrowing down or focusing the question Importance (generate new information or confirm old information in new ways) Feasibility List the possible factors comprising the general question

18 Eg. How well do L2 learners perform speech act? (Broad question) Which speech acts do you want to investigate? What types of L2 learners are you going to choose? In what kinds of situations should the speech events be investigated? Is the research meant to support test development? Material development? Theory development

19 Narrowed Question What is the Chinese EFL learners’ ability to recognize complaint behavior appropriate in an academic university setting? Total performance –recognition of one portion of the total L2 learners – Chinese EFL learners Speech acts – complaint (subcategory) The overall setting – university setting

20 Activity 1 Formulate a research question you are interested in

21 Activity 2 Evaluate the following question: Evaluate the following question: What are the characteristics of the good language learner?

22 From theoretical definition to operational definition Any terms must be operationally defined An operational definition is a clear statement of how you judged or identified a term in your research Good operational definitions can often be drawn from the existing literature

23 Criterion for a good research question The question should Interest us Promise new information or confirm old information in new ways (contribution) Have reasonable scope (feasibility) Have key terms that are clearly defined and operationalized

24 Factors influencing feasibility The amount of time and cost needed Researcher’s prerequisite knowledge Consistent definition of concepts and terms Access to data

25 The purposes of research questions DescriptiveExploratoryExplanatory

26 Step 3: Review the literature on the topic as completely as possible Which questions have been answered satisfactorily? Which seem trivial? Which are too complex to tackle at the moment? Which procedures have been used in answering the questions? What results have been obtained?

27 Step 4: Formulating the research question or the hypothesis A hypothesis is a statement of possible outcome of research. Null hypothesis and alternative hypotheses

28 H 1 and H 2 There is a positive relationship between similarity of relative clause structures in the learners’ L1 and L2 and their performance on L2 relative clauses. There is a negative relationship between similarity of relative clause structures in the learners’ L1 and L2 and their performance on L2 relative clauses.

29 HoHoHoHo There is neither positive nor negative relationship between similarity of relative clause structures in the learners’ L1 and L2 and their performance on L2 relative clauses. In most research reports, the null hypothesis, even if not formally stated, is tested rather than a directional alternative hypothesis. Why?

30 Activity 3 1. Research topic: relationship of language proficiency and spelling test scores 2. How to state and interpret the null and alternative directional hypotheses Null hypothesis: Alternative hypothesis: Directional, positive hypothesis: Directional, negative hypothesis:

31 Null hypothesis: There is no relationship between l. proficiency and spelling scores.  The spelling scores will not vary in relation to the l. proficiency of the Ss. Interpretation of hypotheses

32 Alternative hypothesis: There is relationship …  The spelling scores will vary in relation to the l. proficiency of the Ss.

33 Directional, positive hypothesis: There is a positive relationship between l. proficiency and spelling scores.  Students at higher proficiency levels will obtain higher spelling scores. Students at lower proficiency levels will obtain lower spelling scores.

34 Directional, negative hypothesis: There is a negative relationship between l. proficiency and spelling scores.  Students at lower proficiency levels will obtain higher spelling scores while those with higher levels of l. proficiency will score lower.

35 Activity 4 1. Research topic: gender differences in spelling test scores 2. state a separate null hypothesis for the effect of gender on spelling scores (ignore proficiency) 3. State all the interaction hypotheses (adding in proficiency and L1 membership) H o for gender: H o for gender and L1: H o for gender and language proficiency: H o for gender, L1, and language proficiency:

36 Interpretation H o for gender: There is no effect of gender on spelling scores. H o for gender and L1: There is no interaction of gender and L1 membership on spelling scores.

37 H o for gender and language proficiency: There is no interaction of gender and language proficiency on spelling scores. H o for gender, L1, and language proficiency: There is no interaction of gender, language background, and language proficiency on spelling scores.

38 H o for gender rejected: males and females would show different spelling scores. H o for gender and L1 rejected: there might be a pattern where for females the scores wouldn’t change across the L1 groups while for men they would.

39 H o for gender and language proficiency rejected: there might be a pattern where for females the scores wouldn’t change across the proficiency levels while for men they would. H o for gender, L1, and language proficiency rejected: it might mean, eg, that females of certain L1 groups who are at certain proficiency levels perform differently than others.

40 Answering the question: Internal and external validity Internal validity is the extent to which the outcome is a function of the factor you have selected rather than other factors you haven’t controlled. External validity is the extent that the outcome would apply to other similar situations in the real world.

41 Factors influencing the internal validity Subject selection (size variability, sample size) Maturation (developmental changes in subjects) History (no control of input amount beyond the experiment) Task direction (no clear instructions) Attrition Test effect (the influence of pretest on posttest)

42 Subject selection In order for research to be generalizable, the sample used should be representative of the general population. Methods used to achieve this representativeness include both random sampling and the sample size large enough.

43 Example of subject selection 2 groups of learners have been randomly assigned. Each group contains 10 subjects. After the instructional treatment, a test is given to both groups. The average score for group A is 9.3 while 11.3 for group B. The standard deviations for both are 4.15 and 1.56 respectively. Can we say the method used with group B is superior to that with group A? Why or why not?

44 With this difference between the groups for the SD, we would have to question whether test differences might be due to some other, unidentified factor for the score seem to indicate that the 2 groups are drawn from different populations. The problem for this research is that the subjects are not from a homogeneous population.

45 Maturation Maturation refers to the general developmental changes in subjects during the course of the research esp. in longitudinal studies. The differences may be caused by the treatment, the maturation in cognitive development and short-term memory capability, or a combination of both.

46 History History refers to the possible negative effects of the passage of time on the study. For example, learners may have more access to sources of input or use English more outside of class. In this case, we can’t say the difference in performance is caused by the experimental treatment.

47 Task directions When the instructions are clear, the results may be valid. For example, when a researcher used an instrument of English, the subjects may not understand the instruments accurately.

48 Attrition By attribution we mean that the composition of the population studied may change the longer the study continues. Subjects may lose interest when data collection is made.

49 Test effect If a research begins with a pretest, that test can affect performance during the treatment and on future tests. The test alerts subjects as to what teachers expect them to learn. Then they may pay much attention to it during the course of the research.

50 external validity If a study has no internal validity, then it cannot have external validity. We cannot generalize from the data. When generalizing from the data set, we are both concerned with IV but EV as well, namely how representative the data are for the group(s) to which we hope to generalize.

51 Factors influencing the external validity Population characteristics The effect of the research environment Research effects Data collection methodology

52 Population characteristics Research carried out with university students may not be valid when applied to children. Results from a study of English majors may not be applicable to non-English majors.

53 The effect of the research environment Second language learners may become more motivated simply because they are told that they are participating in a study that will help the researcher understand the process of language learning. This is called the “Hawthorne effect’.

54 Researcher effect The researcher may unintentionally indicate to the subject the kind of response desired.

55 Data collection methodology Data obtained through structured interviews may differ qualitatively from data gathered through a test. It would be difficult to compare the findings of these studies since each approached the problem in a different way.