Needs analysis and evaluation Advisor : Dr. Patricia Su Presenter : Lisa Yin Number : 10022606 Date : April 12th,2012

Aims The key stages in ESP are needs analysis, course (and syllabus) design, materials selection (and production), teaching and learning, and evaluation. These are not separate, linearly-related activities, rather they represent phases which overlap and are interdependent. The simplicity and clarity of figure7.1 is in reality more like figure 7.2 Needs analysis is the process of establishing the what and how of a course; evaluation is the process of establishing the effectiveness.

Needs analysis Figure 7.1 Stages in the ESP process: theory course design teaching - learning assessment evaluation

Needs analysis Figure 7.2 Stages in the ESP process: reality course design teaching - learning assessment evaluation

Needs analysis Needs analysis is neither unique to language teaching nor within language training. However, needs analysis is the corner stone of ESP and leads to a very focused course.

Needs analysis The information obtained from clients and students will only be as good as (a) the questions asked and (b) the analysis of the answers. A crucial point, whenever data is being collected, is to know beforehand what will happen to the raw data and to the information derived from it. As ESP practitioners we need to know exactly what we are trying to find out and what we will do with the answers before we start. (Berwick,1989:62)

What is meant by needs? A confusing plethora of terms exists: needs are described as objective and subjective, perceived and felt , target situation /goal-oriented and learning, process-oriented and product-oriented ; in addition, there are necessities, wants and lacks. These terms have been introduced to describe the different factors and perspectives which have helped the concept of needs to grow. Each of these terms represents a different philosophy or educational value. Briefly, objective and perceived needs are seen as derived by outsiders from facts, from what is known and can be verified, while subjective and felt needs are derived from insiders and correspond to cognitive and affective factors.

What is meant by needs? Target situation analysis(TSA) Learning situation analysis(LSA) Present situation analysis(PSA ) Means analysis (Holliday and Cooke, 1982:133) A TSA includes objective, perceived and product-oriented needs; An LSA includes subjective, felt and process- oriented needs; A PSA estimates strengths and weaknesses in language, skills, learning experiences.

What is meant by needs? To establish a workable course design, means analysis is suggested (Holliday and Cooke, 1982:133)as an adjunct to needs analysis. Means analysis looks at the environment in which a course will be run or, as in the original metaphor that generated the term, the environment in which a project will take root, grow healthily and survive. The two key factors considered for Means analysis are the classroom culture and the management infrastructure and culture.

A current concept of needs analysis Needs analysis in ESP now encompasses determining: professional information about the learners : the tasks and activities learners are/will be using English for – target situation analysis and objective needs personal information about the learners : factors which may affect the way they learn such as previous learning experiences, cultural information, reasons for attending the course and expectations of it, attitude to English – wants, means , subjective needs English language information about the learners : what their current skills and language use are – present situation analysis – which allows us to assess (D) the learners’ lacks : the gap between (C) and (A) – lacks

A current concept of needs analysis E. Language learning information : effective ways of learning the skills and language in (D) – learning needs F. professional communication information about (A) : knowledge of how language and skills are used in the target situation – linguistic analysis, discourse analysis, genre analysis G. What is wanted from the course H. Information about the environment in which the course will be run – means analysis

A current concept of needs analysis environmental situation Personal information about learners Language information about target situations Learners’ lacks learners’ needs from course Language learning needs how to communicate in the target situation professional information about learners What needs analysis establishes

Matching needs analysis to situation A course outline, materials and other resources can be in place before teaching begins. We must distinguish between overall needs and course needs. Course needs : What do you need/ want from the course? Overall needs : What do you need English for?

Evaluation What is evaluation? There are many definitions of evaluation ; fundamentally evaluation is asking questions and acting on the responses. We accept the following : evaluation is a whole process which begins with determining what information to gather and ends with bringing about change in current activities or influencing future ones. Evaluation must be more than collecting and analyzing data: to have value the evaluation process must include action. Evaluation is usually described as formative or summative.

What is evaluation? Formative evaluation Summative evaluation Evaluation is usually described as formative or summative. Formative evaluation Summative evaluation which takes place during the lifetime of an activity(a course) and the findings help to shape the course during its life-time. is typically undertaken at intervals and will consist of a series of ‘mini-evaluations’. Testing the arrangement of lessons in a primer before its publication Collecting continuous feedback from participants in a program in order to revise the program as needed takes place at (or after) the end of an activity and so does not influence that version of the activity. Its purpose is to assess impact and to provide information that can be fed into repeat versions or related activities. Determining attitudes and achievement related to using a primer after it has been used in a training course Collecting data on the impact of a program operating in a community for a period of time

What is evaluation? Evaluation can be both qualitative and quantitative. Evaluation can be very threatening; it suggests change and change is often resisted. Evaluation is a very constructive and powerful activity and a very stimulating one. Evaluation will also show weaknesses or features that were just not suitable for the particular group of learners. A good evaluation emphasizes the successes and discusses less successful aspects. It also addresses the crucial how and why issues. Knowing how well something has worked is not significant on its own. Understanding why will enable us to repeat success and avoid the less successful.

What do we evaluate? Evaluation in ESP situation is concerned with the effectiveness and efficiency of learning. Evaluation results can be used to influence decisions and bring about long-term change. One-off courses Durable courses The important questions are those asked part-way through Focus on change that is feasible and immediately implementable. Evaluation questions may be asked for a range of different purposes.

What do we evaluate? -1 Some of the questions to ask before an evaluation are: Audience and purpose Who are the stakeholders? The term ‘stakeholder’ is used to cover all those who have an interest or concern with the course. The client who requests the course, th learners and the teachers are the main stakeholders but there can be others (sponsors, organisers). What do you want to evaluate? What do you want to change?

What do we evaluate?-2 Criteria for evaluation What are the objectives you are evaluating against? In some situations Criteria for analysis of results What will you do with the answers? What can you change? What requires the authority of others? And what will convince them? Sources of information Who can provide useful information? When Would it be appropriate for them to do that?

Collecting data for needs analysis and evaluation purposes Who collects the data? For both needs analysis and evaluation, outsiders or insiders could be involved. Outsiders, they do not know the situation and the environment so they may miss or misinterpret data. Insiders will have a feel for the situation but can be too close and involve, or lack expertise. A known, respected outsider working together with insiders is one alternative (Alderson and Scott, 1992:36/7) The outsiders can be others in the same institution who are not involved with the course that is being evaluated. Complete outsiders are most likely to be used both for needs analysis and evaluation on large-scale projects with external funding.

Who provides the data and how? Checklists and questionnaires Checklists are narrower in scope and more commonly used for a qualitative feel. They can determine facts or attitudes. Of the attitude scales the easiest to construct is an adjective checklist. The Likert scale consists of statements that respondents agree or disagree with. Rating scales are easier to use and useful for broad distinctions.

Structured interviews Structured interviews consist of questions which have been carefully thought out and selected in advance. Structured interviews should be recorded so that the interviewer can really listen rather than take lots of notes. The art is to gain the maximum relevant information in the minimum time. Key skills for interviewers are active listening, summarising and asking open questions.

Observation Observation can cover a range of activities from watching a particular task being performed to shadowing individuals at work. EAP examples of observation for needs analysis include sitting in on subject lectures or practical sessions. Observations have led to an understanding of how and when English and the L1 are intertwined, of the code- switching which often goes on. In EOP situations, any relevant work processes may be observed. Most observation or shadowing is for Target Situation Analysis(TSA) purposes but it could be for the Present Situation Analysis (PSA) of a particular individual.

Observation Most observation for evaluation is of classroom activity , but it could also include seeing how well a learner was coping with using the language in their work or studies. Classroom observation requires careful groundwork and handling. After the session, observer and teacher should share their perceptions of what was happening. The early comments must focus on positive features. Beginning with negative comments sets up the wrong perspective.

Analysis of authentic texts The texts can be written documents or audio and video recordings of events such as lectures, meetings, telephone interactions, classroom activities. The analysis may be for TSA purposes, to determine the key linguistic features of a communicative event or genre that is new to us. Authentic texts are invaluable for learning about real and carrier content.

Assessment Assessment includes formal and informal judgements of students’ performance and progress through classwork, assignments and tests. Testing or assignments may form part of a pre-course PSA or evaluate progress. Issues of assessment and testing are covered in chapter 11.

Discussions Record keeping Discussions can pave the way in both needs analysis and evaluation to other methods such as interviewing and observing or be an end in themselves. Record keeping What is needed in record keeping is systematicity: records must be made immediately.

How do we analyse the data? First, the raw data must be converted into information. With small amounts of quantitative data manual methods can be employed to process the data into information. Jones(1991) described how Principal Component Analysis(PCA) By using PCA, which is a type of correlational analysis that can resolve data into underlying factors, he was able to identify clusters of variables. LANA(LAnguage Needs Analysis) is another tool intended either for individual needs analysis or as part of an audit. Needs analysis must result in an understanding of a target situation such that we, the ESP practitioners, could be efficient communicators in it.

What happens to the results? The feedback may be how a text is to be used, how the design of a course component has been influenced, the overall results of a mini- evaluation. Feedback is good PR(public relations), good for the quantity and quality of future cooperation.

Summary Behind successful ESP courses is a continuous process of questioning. Ascertaining what a course should contain, how it should be run; B. Checking throughout how valid the original answers were and how effective the ideas they led to are; C. Discovering what works best and why. Initial needs analysis On-going needs analysis Formative evaluation Summative evaluation set objectives revise objectives inform future determine approach modify teaching and materials justify measure

Thank you !