Washback of BiH STANAG 6001 test Major Drazen Malesevic BiH STANAG 6001 team  

Outline Definition of washback Terms in usage Micro and macro level of impact Impact on educational system (Impact of BiH STANAG 6001 test on ELT in AF BiH) Micro level of washback Types of washback How to achieve beneficial washback Conclusion  

To refer to the theory of washback To present the case of BiH PURPOSE To refer to the theory of washback To present the case of BiH To analyze how to achieve beneficial washback  

Definition of washback Different definitions Different terms washback, backwash, impact   There is no unique definition of washback. It is perfectly clear for all authors that there is effect of testing, but they do not agree on the scope of that impact. Some argue that it only affects individuals in the process of teaching and learning while others claim it affects not only the individuals in classroom but wider population, in some cases the whole nation. There is no unique usage of the term when we analyze work of different authors. Some of them use washback while others use back-wash. For both terms they find enough reasons to be valid for usage and it is not wrong to use any of them. In the case of this presentation we will use the term washback. In some books, we can find these differences in the term usage when it comes to the extent of the effect. If we consider the effect of test on wider population other than classroom, in that case we talk about IMPACT but if we talk only about individuals in classrooms then we talk about washback.

The effect of testing on instruction WASHBACK: The effect of testing on instruction From: Dictionary of Language Testing, 1999 Very simple definition of washback can be found in Dictionary of Language testing, 1999

IMPACT “Any of the effects that a test may have on individuals, policies or practices, within the classroom, the school, the educational system or society as a whole” Wall (1997)   Dianne Wall defines Impact as …………

“the effects of tests on teaching and learning” Washback “the effects of tests on teaching and learning” (Wall, 1997, p. 291)  Dianne Wall defines washback as …………

Macro and micro level of impact Macro level impact Impact on society and educational system Micro level impact Impact on individuals involved in teaching and learning process (Bachman and Palmer)   There is also an interesting classification of the impact on Macro and Micro level done by Bachman and Palmer in their book Language Testing in Practice. When they talk about the impact on macro level they think of the impact of a test on education system and society but when it comes to micro level they talk about individuals involved in teaching and learning

IMPACT ON SOCIETY University admission exam in Liberia - None of 25 thousand students passed (BBC NEWS, 26 August 2013) An interesting example of how test can have an impact on the whole society is an example of this year’s admission exam to university in Liberia. None of 25 thousand students passed the exam. This certainly has an impact on the society. This year, in Liberia there will be no students of the first year at the university. It is not a problem only for the students and their parents, it is a problem for the whole state and society.

IMPACT ON EDUCATIONAL SYSTEM (Impact of BiH STANAG 6001 test on ELT in AF BiH) ELT since 1998 BiH STANAG 6001 since 2009 The same objective (competent English language speaker properly assessed to operate effectively in the English language environment) When it comes to the impact of test on education system I will refer here to the example of introducing BiH STANAG 6001 test and its impact on the ELT system in AF BiH.

ELT in AF BiH Introduced in 1998 Two different programs in the beginning Four levels of the course (elementary, pre-intermediate, intermediate and upper-intermediate) The length of all courses was 12 weeks After introducing STANAG 6001 five levels of the course (advance level of the course introduced) Since 1998 we have had ELT in BiH. It started by introducing two different programs run by two different organizations. One program sponsored by DLI and the other sponsored by the British Council. Later on they merged into one program run by one organization within AF of BH. There were four levels of course each 12 weeks in length. The levels of the course were elementary, pre-intermediate, intermediate and upper-intermediate Initial results of the STANAG test in AF BiH were promising but it was only at the beginning. After several sessions the results became worse. After thorough analyses of our candidates we have concluded that in the initial phase our candidates were mainly those who were long waiting for STANAG and with certain international experience. After them, we started to test candidates whose only experience was language course. Their results were not satisfactory. The proposal was to introduce one more level of the course to offer the potential candidates better conditions for better results on STANAG

ELT in AF BiH Curriculum was changed after introducing STANAG 6001 The length of the course remained the same, 12 weeks More exercises were added to the curriculum in order to improve certain skill (listening, speaking, reading, writing) After introducing STANAG, some changes were made to the curriculum of the course. The length remained the same but some changes within the curriculum were made Specific exercises to the curriculum based mainly on the books from DLI program were added to the curriculum. The most critical skills were speaking and writing. Reading and listening were added to the curriculum as well. The differences between the old and new curriculum can be seen on the next two slides

As you can see here on the slide As you can see here on the slide. The curriculum was based only on the content of DLI books. After introducing SSTANAG 6001 test, some things changed and on the next slide, we can see them

Now, as we can see, the curriculum is a combination of the DLI books content and activities related to certain skills. In the morning, curriculum is based on the books from DLI program but after the lunch break, instructors have freedom to chose materials for improvement of a certain skill. It is only specified which skill is on which day. Any supplementing materials can be used to enhance the given skill. In this case, we can see supplementing materials from IELTS, New Headway, or some other materials presented by instructor After introducing the new curriculum, results on STANAG were better, students were more satisfied.

Micro level of washback test takers Officers, NCOs, civilians, MoD officials instructors Another important aspect of washback is an impact of a test on micro level. The impact on individuals involved in the process of teaching and learning. But, here I would like to emphasize a very important issue of the stakes of STANAG testing in armed forces For test takers it is to be or not to be in their carriers. Most carrier paths are nowadays related to international cooperation and without good language command and proper SLP it is impossible to advance though carrier. So proper institutional education is necessary for them to acquire the desired level of language Having that in mind we have to think about people who are due to deliver English language training in a proper way so the students are satisfied and their results are on a satisfactory level. Also, it is worth mentioning that methods of teaching and instructing are affected by the test. Instead of following methodology of DLI books, instructors have to adapt to the new teaching methods related to the new supplementing materials.

The stakes of STANAG 6001 test Selection of personnel for: promotion to higher rank higher positions military diplomatic representatives PSO missions UN observer missions Schools and courses abroad (Book of rules) In this case, in the AF BiH one of the criteria for selecting personnel for higher positions is the result on STANAG 6001 test. The higher result on STANAG the better chances to be appointed on a higher position. The same thing is for all other things in bullet points specified here, military diplomatic representatives……….Probati ubaciti pavilnike ovdje

The stakes of STANAG 6001 test Promotion to higher rank in Montenegro: Officers must have SLP 2222 NCOs must have SLP 1111 (Law of service) How important STANAG can be for officers and NCOs, shows an example of the Law of service in the AF of Montenegro. In order to be promoted to a higher rank, an officer in Montenegro must have at least SLP 2222 and for NCOs it is at least SLP 1111. Without the English language, there is no promotion to higher rank

Types of washback Harmful Beneficial Now, I would like to say something about types of wash-back. As you can see on the slide, it can be harmful and beneficial. Of course we all tend to have beneficial washback. It is beneficial if a test has positive impact on learning process, otherwise it is harmful

How to achieve beneficial washback Test the abilities whose development we want to encourage Sample widely and unpredictably Hughes, p. 53-57 Hughes, in his book Testing for language teachers explains how to achieve beneficial washback. First, he states that it is necessary to test the abilities we want to encourage. In the case of STANAG testing it is obvious that we test four skills (listening, speaking, reading and writing) and the objective of the course should be encouragement of those four skills. If we test objectives of a course we will achieve positive was-back. In the case of STANAG, objectives of the course should be derived from STANAG descriptors. When it comes to sampling, it is obvious we cannot test everything from our specifications, but our sample of the test should represent the wide scope of what is given in our specifications

How to achieve beneficial washback Use direct testing Make testing criterion referenced Hughes, p. 53-57 Direct testing is necessary if we want to have a valid test. For example, when testing the skill of writing it is necessary to get our candidates to write. Our tasks for writing have to be authentic and have to be derived from STANAG descriptors. For large scale testing it is not practical to use direct test because each composition should be rated at least by two raters. It is more practical to test writing by multiple-choice tests but it doesn’t contribute to beneficial washback. STANAG test is a criterion referenced test. With this type of test we measure performance of a candidate against criterion, in the case of STANAG it is the scale of levels ranging from Level 0 to Level 5, supplemented by descriptors for each level. In this case all participants in the process of learning, teaching and testing are aware of the requirement for each level

How to achieve beneficial washback Base achievement tests on objectives Ensure the test is known and understood by students and teachers Documents related to testing on MoD site Seminars related to testing organized for teachers Hughes, p. 53-57 In this case there is no need to think about our objectives. Objectives of STANAG are derived from descriptors. We cannot base our STANAG test on content of any course. In that case we would not have a professional test, it would be something different. As opposed to criterion referenced tests there are norm referenced tests in which we measure performance of a candidate against performance of other candidates our candidates and their teachers have to be familiar with our tests. They have to be aware of what to expect on test. There should be no surprises on test, we do not want to trick our candidates, we want to measure their proficiency in language. Everything from specifications that concerns our candidates should be available for them (test format, test methods, types of texts, length of texts, etc.) Some samples of test could be also helpful and they could contribute to beneficial washback. In our case, we have all our documents related to STANAG testing posted on our MoD site. There are specifications, book of rules, some samples and other documents related to STANAG on that site. In the phase of preparing a list of candidates for testing, test takers are reminded about that at least a month ahead of testing session

How to achieve beneficial washback Where necessary, provide assistance to teachers Counting the cost Practicality Hughes, p. 53-57 Our teachers are familiar with our test as well. In our case, we have regular communication with our teachers, STANAG team members participate on seminars for instructors to inform them on changes related to STANAG testing At the beginning of STANAG testing in our country, we organized a seminar tailored to their needs, the seminar was lectured by Jana Vasilj-Begovic, Peggy Garza and MaryJo Di Biase. It was held in Sarajevo Very often it is costly to create a test, it is time consuming, It is expensive to have professionals only for developing tests, administration of a test is also expensive, but at the end of the day it is more expensive not to test at all. In that case we would sent our personnel to missions without a clue of their language knowledge, which could sometimes have fatal consequences.

STANAG test has a huge impact on: After introducing STANAG 6001: Conclusion STANAG test has a huge impact on: individuals in AF BiH ELT system After introducing STANAG 6001: New level of course introduced Curriculum changed

Is STANAG test mandatory after course? Conclusion Is STANAG test mandatory after course? It is not, but usually students obtain better results It is not dependent on any course content Are they interwoven? In a way, they are. (They both have the same objective, a competent and properly assessed language user) They both have to keep their independence (even though, they are in a way interwoven)

Table of references: Lyle F. Banchman and Adrian S. Palmer, Language Testing in Practice: Designing and Developing Useful Language Tests, Oxford University Press; Arthur Hughes, Language Testing in Practice, Cambridge University Press; Harold S. Madsen, Techniques in Testing; Liying Cheng and Yoshinori Watanble, Washback in Language Testing: Research Contexts and Methods, LAWRENCE ERLBAUM ASSOCIATES, PUBLISHERS 2004 Mahwah, New Jersey London; J. CHARLES ALDERSON and DIANNE WALL, Does Washback Exist? Lancaster University BBC News, http://www.bbc.co.uk/news/world-africa-23843578 Book of rules in AF BiH (related to promotion, carrier, education etc.)

Questions?