1. 1 TESL 4340 -- Evaluating CALL Packages:Curriculum/Pedagogical/Lingui stics Dr. Henry Tao GUO Email: henryguo@uic.edu.hkhenryguo@u Office: B 418

2. 2 Outline Part one: Language program evaluation (Lynch 1996: 1-9) Part two: Evaluating CALL( Chapelle 2001:44-94)

3. 3 Part one : Language program evaluation Critical issues Context-adaptive model (CAM)

4. 4 Evaluation Evaluation here is defined as the systematic attempt to gather information in order to make judgments or decisions.

5. 5 As such, evaluative information can be gathered through different methods such as Observation Questionaire Interview …

6. 6 Why CALL evaluation? Internally, to better a program Externally, to justify program funding

7. 7

8. 8 Context Adaptive Model (CAM) Who needs the information? And what are the goals of evaluation? What are the roles of evaluators? Contextual questions Which aspect of the program should the evaluation focus on? Data collection and analysis Evaluation report (Adapted from Lynch 1990a)

9. 9 Who needs the information? Audience Who is requesting the evaluation? Who will be affected by the evaluation? Stakeholders (or clients) & Peripheral audience

10. 10 What are the goals of evaluation? Why is the evaluation being conducted? What information is being requested and why? The goals of evaluation may be determined by audience

11. 11 What are the role of evaluator? The role of evaluator is determined by the particular audience and their goals  Someone providing consultation  An expert standing in judgment  A collaborator in program development  A decision-making facilitator

12. 12 Hiring external evaluators for greater objectivity or internal evaluators for improving the curriculum?

13. 13 Contextual questions 1.Is there a comparison group? 2.Are there reliable and valid measures of language skills? 3. Are there various types of evaluation expertise? 4. Timing of the evaluation 5. How students are admitted into the program?

14. 14 6. What the program students are like 7. What the program staff are like 8. Size and intensity of the program 9. Instructional materials and resources 10. Perspective and purpose of the program 11. Social and political climate

15. 15 Preliminary thematic framework (PTF) Where should the evaluator begin? What aspects of the program should evaluator investigate in detail?

16. 16 An example of PTF developed for EST reading program 1.Effects of focusing instruction on reading only 2. Effects of focusing instruction on reading skills and strategies 3. Effects of using authentic reading texts 4. Feasibility of using students’ mother tongue versus target language for instruction

17. 17 5. Availability of classrooms 6. Feasibility of using a ‘modified adjunct model’ approach 7. Feasibility and effects of conducting classroom-centered research 8. Levels of student proficiency in English upon entering the program (Adapted from Lynch 1990a)

18. 18 Data collection design/system What type of data need to be gathered- quantitative, qualitative or both? What will be the best method for gathering the data? The contextual questions can be used to determine which types of data collection design is possible.

19. 19 Data collection and analysis How is data collection to be conducted? How is the results to be interpreted?

20. 20 In the case of quantitative designs, have the assumptions of the design and statistical models been met? In the case of qualitative design, have the procedures for data gathering been portrayed accurately, and have alternative interpretations of the data been pursued?

21. 21 Evaluation report The evaluator must be sensitive to the audience and goals of the evaluation The social and political climate dimension of the context needs to be considered carefully at this stage

22. 22 The critical issue is how to communicate the findings of the evaluation honestly and successfully. Prepare multiple reports or express the findings in different ways, depending on the intended audiences.

23. 23 Part two : Evaluating CALL The problem of instructed SLA Principles for CALL evaluation Judgmental evaluation of CALL Empirical evaluation of CALL

24. 24 The problem of instructed SLA Cognitive conditions for SLA Social-affective conditions for SLA Other factors

25. 25 Cognitive conditions for SLA  Choose a range of target structures  Choose tasks which meet the utility condition  Select and sequence tasks to achieve balanced goal development  Maximize the chances of focus on form through attentional manipulation  Use cycles of accountability

26. 26 Social-affective conditions for SLA  Willingness to communicate (WTC) The desire to communicate with a particular person Communicative self confidence at that particular moment Interpersonal motivation Intergroup motivation

27. 27 Self confidence Intergroup attitudes(e.g., integrativeness) Social situation(i.e.,features of context affecting communication) Communicative competence Intergroup climate personality

28. 28 Other factors  Individual differences in cognitive characteristics of learners due to, for example, age or cognitive style.  Learning situation such as the effects of task choice on teachers and learners and others who may be involved with the task  Practical factors such as the available resources

29. 29 Principles for CALL evaluation Evaluation of CALL is a situation specific argument Judgemental anlaysis of software and planned tasks and empirical analysis of learners’ performances

30. 30 Criteria should come from theory and research on second language teaching Criteria should be applied in view of the purpose of the task Language learning potential should be the central criterion in evaluation of CALL

31. 31 Judgmental and empirical analyses

32. 32 Criteria from theory and research on SLA Language learning potential Learner fit Meaning focus Authenticity Positive impact practicality

33. 33

34. 34 Criteria applied based on task purpose These criteria for CALL appropriateness need to be applied in view of the purpose of a CALL task. Moreover, tasks may have different purposes at various stages of instruction (Doughty & Williams,1998). Whatever the goal of the CALL task, however, evaluation of the task requires that it have a stated purpose.

35. 35 The centrality of language learning Even though the importance of each of the six criteria may vary depending on the purpose of the tasks, language learning potential should be considered the most critical for CALL activities

36. 36 Judgemental evaluation of CALL

37. 37

38. 38 Cases(Chapelle 58-67) Computer-assisted classroom discussion Microworld Text analysis Storyboard Concordancing

39. 39 Summary

40. 40 Empirical evaluation of CALL Why? ‘Students are often doing something very different from what [ language teachers ] assume they are doing’ ( Hosenfeld,1976:123). In other words, it is necessary to identify the observable data that provide evidence of CALL qualities.

41. 41 How? Look for evidences

42. 42

43. 43 Language learning potential Focus on form Modified interaction Modified output

44. 44 Learner fit Level of linguistic difficulty Individual difference

45. 45 Meaning focus Effects of meaning-based instruction Assessing engagement with meaning

46. 46 Authenticity Comparing CALL with non-CALL activities

47. 47 Positive impact

48. 48 Practicality Given the role of resources for the success of CALL, 1) some formal mechanism needs to be in place to monitor adequacy, and 2) an argument about CALL appropriateness should include a statement about sufficiency of resources.

49. 49 Conclusion Evidence concerning the six ideal qualities of a particular CALL task needs to be combined to form an evaluative argument about the appropriateness of a CALL task for particular learners at a given point in time.

50. 50 Integrated research is needed to examine the types of CALL activates to seek evidence about CALL appropriateness in particular settings.

51. 51 From such research, implications can be drawn for specific contexts, and eventually research on particular types of activities might be generalised in a way that can inform general principles of CALL.