Research Methods of Applied Linguistics and Statistics (3) Types of research Constructing Research Designs

Research Types Data sources: Data sources: primary, secondary primary, secondary Objectives: Objectives: descriptive, exploratory, correlational, explanatory descriptive, exploratory, correlational, explanatory Time for data collection: Time for data collection: cross-sectional, longitudinal (panel, trend) cross-sectional, longitudinal (panel, trend) Data types: Data types: qualitative, quantitative (survey, experimental) qualitative, quantitative (survey, experimental)

Primary: studies based on primary, or original, data sources, such as classroom observations or real students, or their test scores, or their responses to a questionnaire. Primary: studies based on primary, or original, data sources, such as classroom observations or real students, or their test scores, or their responses to a questionnaire. Secondary: studies based on secondary sources such as other researhers ’ books and articles, including library research and literature reviews. Secondary: studies based on secondary sources such as other researhers ’ books and articles, including library research and literature reviews.

Descriptive: studies involving the collection of data in order to test hypotheses or to answer questions concerning the current status of the subjects of the study. Descriptive: studies involving the collection of data in order to test hypotheses or to answer questions concerning the current status of the subjects of the study. Exploratory: studies conducted into an issue or problem where there are few or no earlier studies to refer to. The focus is on gaining insights and familiarity for later investigation. Exploratory: studies conducted into an issue or problem where there are few or no earlier studies to refer to. The focus is on gaining insights and familiarity for later investigation.

Correlational: The basic question for descriptive research is - "What are the values of a number of variables for a given sample of subjects. The basic research question for correlation research is - What is the relationship between two or more variables for a given set of subjects. Notice that we said relationship between variables and not the effect of one variable on another variable. Correlational: The basic question for descriptive research is - "What are the values of a number of variables for a given sample of subjects. The basic research question for correlation research is - What is the relationship between two or more variables for a given set of subjects. Notice that we said relationship between variables and not the effect of one variable on another variable. In descriptive research we are just describing our subjects in terms of one or more variables, while in correlational research we are looking at the relationship between the variables. In descriptive research we are just describing our subjects in terms of one or more variables, while in correlational research we are looking at the relationship between the variables.

Explanatory: studies conducted in order to explain any behaviour in L2 learning and teaching. It could be done by using questionnaires, group discussions, interviews, etc. Explanatory: studies conducted in order to explain any behaviour in L2 learning and teaching. It could be done by using questionnaires, group discussions, interviews, etc. Cross-sectional: studies in which groups of participants of different ages are observed and compared at a given time Cross-sectional: studies in which groups of participants of different ages are observed and compared at a given time

Longitudinal: It is research that studies a person or group over a set period of time, normally to track the effect of some variable and as such permits causal pathways to be determined. For example, when trying to discover the effect of language aptitude on L2 proficiency, a researcher may track 50 children over a 30 year period in order to find a common factor between those children who are quicker and effective L2 learners. Longitudinal: It is research that studies a person or group over a set period of time, normally to track the effect of some variable and as such permits causal pathways to be determined. For example, when trying to discover the effect of language aptitude on L2 proficiency, a researcher may track 50 children over a 30 year period in order to find a common factor between those children who are quicker and effective L2 learners.

Constructing Research Designs Determining the design Determining the design Classification of designs Classification of designs

Research Design 1: Quantitative / Qualitative Quantitative research is empirical research involving analysis of numerical data Its aim is to determine the relationship between one thing (an independent variable) and another (a dependent or outcome variable) in a population. Quantitative research designs are either descriptive (subjects usually measured once) or experimental (subjects measured before and after a treatment). A descriptive study establishes only associations between variables. An experiment establishes causality. Quantitative research is empirical research involving analysis of numerical data Its aim is to determine the relationship between one thing (an independent variable) and another (a dependent or outcome variable) in a population. Quantitative research designs are either descriptive (subjects usually measured once) or experimental (subjects measured before and after a treatment). A descriptive study establishes only associations between variables. An experiment establishes causality.

Qualitative research is empirical research involving analysis of data such as words (e.g., from interviews), pictures (e.g., video), or objects (e.g., an artifact). Its aim is building theory. It is generally inductive in approach, is based originally on the naturalistic assumption that reality is mind dependent (i.e., can only be known as it is interpreted and has "meaning" for the observer), is usually of single-subject design, and generally deals with nominal data Qualitative research is empirical research involving analysis of data such as words (e.g., from interviews), pictures (e.g., video), or objects (e.g., an artifact). Its aim is building theory. It is generally inductive in approach, is based originally on the naturalistic assumption that reality is mind dependent (i.e., can only be known as it is interpreted and has "meaning" for the observer), is usually of single-subject design, and generally deals with nominal data

Philosophical base Phenomenology (quali.)Logical positivism (quan.) Data from personal involvement Data from experiment SubjectiveObjective EmpathyDistance Data in the form of string Data in form of number

Methodology SyntheticAnalytic From part to wholeFrom whole to part Internal structureExternal structure Process-orientedProduct-oriented Dynamic reality hypothesis Static reality hypothesis

Purpose InductionDeduction Hypothesis generatingHypothesis confirming Exploratory, descriptiveInferential, confirmatory, explanatory Result of pattern or hypothesis Formation of theory

Features for both designs High validity, low reliability Low validity, high reliability Preference of content to form Preference of form to content Low generalizing capacity High generalizing capacity High explanatory power Low explanatory power

Mixed design: Balanced

Mixed design: Unbalanced

Literature review of quantitative and qualitative studies by Lazaraton (2000) Current Trends in Research Methodology and Statistics in Applied Linguistics Current Trends in Research Methodology and Statistics in Applied Linguistics

Research Design: experimental research An experimental study is a study in which the researcher manipulates one or more independent variables and measures their effect(s) on one or more dependent variables while controlling the effect of extraneous variables. An experimental study is a study in which the researcher manipulates one or more independent variables and measures their effect(s) on one or more dependent variables while controlling the effect of extraneous variables.

Major components of experimental research The population The population A comparison which involves at least 2 groups or 2 conditions. A comparison which involves at least 2 groups or 2 conditions. The treatment The treatment A treatment in which one or more independent variables are manipulated. A treatment in which one or more independent variables are manipulated. The measurement of the treatment The measurement of the treatment The measurement of dependent variable(s) as the result of the treatment. The measurement of dependent variable(s) as the result of the treatment.

Letters standing for components X: an experimental treatment (teaching method, exposure to a material … ) X: an experimental treatment (teaching method, exposure to a material … ) － : no treatment － : no treatment O: measurement of the effects of the treatment or a test O: measurement of the effects of the treatment or a test R: randomization, or the random assignment of subjects to groups in order to control for extraneous variables. R: randomization, or the random assignment of subjects to groups in order to control for extraneous variables.

Types of experimental studies Pre-experimentalQuasi-experimentalTrue-experimental One-group posttest only study: X O 1 Posttest only nonequivalent groups: EG: X O 1 CG: O 2 Posttest only equivalent groups EG: R X O 1 CG: R O 2 One-group pre/post test study: O 1 X O 2 Pre/post test nonequivalent groups: EG: O 1 X O 2 CG: O 3 O 4 Pre/post test equivalent groups: EG: R O 1 X O 2 CG: R O 3 O 4 No control group; no randomization a control group; no randomization a control group + randomization

Pre-experimental: single group design One-shot design One-shot design One group pretest-posttest design One group pretest-posttest design Time-sampling design Time-sampling design

Example for one-shot design An English teacher used the immersion teaching method for teaching English communicative use. After 3 weeks, a test is administered to the class. The class performs well on the test. An English teacher used the immersion teaching method for teaching English communicative use. After 3 weeks, a test is administered to the class. The class performs well on the test. X O X O It is also called a pilot study design It is also called a pilot study design

One group pretest-posttest design It is also called repeated measures. It uses the subjects as their own controls and eliminates the need for a control group design. It is also called repeated measures. It uses the subjects as their own controls and eliminates the need for a control group design. Strength: It is efficient because some extraneous variables, attrition and loss of subjects controlled. Strength: It is efficient because some extraneous variables, attrition and loss of subjects controlled. Weakness: validity might be affected by history, attrition and maturation. Weakness: validity might be affected by history, attrition and maturation.

Time-sampling design It is also referred to as time-series designs. It is also referred to as time-series designs. O 1 O 2 O n X O 4 O 5 O n O 1 O 2 O n X O 4 O 5 O n O 1 X O 2  O 3 -- O 4  O 5 X O 6  O 7 -- O 8 O 1 X O 2  O 3 -- O 4  O 5 X O 6  O 7 -- O 8

Quasi-experimental: posttest only nonequivalent groups EG: X O 1 CG: O 2 O 1 -O 2 The treatment effect could also be attributed to a series of extraneous variables.

Posttest/posttest nonequivalent groups EG: O 1 X O 2 CG: O 3 O 4 (O 2 -O 1 ) – (O 4 -O 3 ) (O 2 -O 1 ) – (O 4 -O 3 ) Without randomization, the confounding effects cannot be fully controlled. Without randomization, the confounding effects cannot be fully controlled.

True-experimental: posttest only equivalent groups EG: R X O 1 CG: R O 2 O 1 -O 2 The design assumes that the 2 groups are identical on all relevant features, but it is still sensitive to selection bias, mortality, Hawthorn effects.

Pre/post test equivalent groups EG: R O 1 X O 2 CG: R O 3 O 4

Factorial Designs Factorial designs are similar to the true experimental designs and include all of the elements found in those designs, such as randomization, pre- and post-testing, and treatments. The difference is that the effects of independent variables and a moderating variable may be tested at the same time. Factorial designs are similar to the true experimental designs and include all of the elements found in those designs, such as randomization, pre- and post-testing, and treatments. The difference is that the effects of independent variables and a moderating variable may be tested at the same time.

Example for a factorial design In a study to test for the effect of language laboratory training on pronunciation, it is also decided to measure for the effect of such training on learners of different proficiency levels, as well as the effect of different kinds of language laboratory practice. In a study to test for the effect of language laboratory training on pronunciation, it is also decided to measure for the effect of such training on learners of different proficiency levels, as well as the effect of different kinds of language laboratory practice. Types of practice (contextualized / decontextualized drills), proficiency levels of the learner (beginners / advanced learners ) Types of practice (contextualized / decontextualized drills), proficiency levels of the learner (beginners / advanced learners )

Exemplary Figure for Factorial Design

Experimental study by Robb, et al. (1986) Salience of feedback on error and its effect on EFL writing quality Salience of feedback on error and its effect on EFL writing quality Salience of feedback on error and its effect on EFL writing quality Salience of feedback on error and its effect on EFL writing quality Types of feedback: complete correction, coded feedback; yellow highlighting of errors, listing the number of errors Types of feedback: complete correction, coded feedback; yellow highlighting of errors, listing the number of errors The compositions were graded on 19 features which were collapsed into 7 categories. The compositions were graded on 19 features which were collapsed into 7 categories.