Because organizational environments are constantly demanding that employees learn continuously, it is crucial that training programs be effective in reaching every individual. According to Kirkpatrick (1976), there are four levels of evaluation training: reaction, learning, the performance in the job and results. Trainee reaction to training is a very relevant variable, usually associated with high levels of impact of training at work. Despite such importance, valid and reliable trainee reaction measures are still rare (Abbad, Gama &Borges-Andrade, 2000). According to Borges-Andrade (2002), studying reaction usually involves the use of a questionnaire with a Likert-like scale and at least one open question, which demands content analysis. Usually, qualitative and quantitative analyses present high degrees of correspondence. This study presents two experiences in examining trainee reaction to distance training programs. The first one consists of the validation process of a questionnaire that measures trainee reaction to the instructional procedures used in two distance education courses. The second one describes the use of qualitative methods in the formative evaluation of a web-based training program. Evaluating reaction to distance training programs: combining quantitative and qualitative methods ABSTRACT METHOD This study is an attempt to trigger a discussion about the combined use of both quantitative and qualitative methods to evaluate trainee reaction to training programs. Recent qualitative research conducted in Brazil resulted in a valid and consistent questionnaire containing 12 items that evaluated training reactions to traditional procedures and procedures based on new information technology. Research on trainee reaction has also been done with qualitative methods, such as the content analysis of learners’ answers to open questions related to strategies, content, layout and other aspects of web-based courses. Once both methods provide researchers with precious, useful information, the combined use of them is suggested. As Borges-Andrade (2002) points out, the process of creating an instrument for collecting quantitative data may also include qualitative tools (interviews and open-questioned surveys). Therefore, it is suggested that qualitative and quantitative methods are used together to evaluate reaction. Study 2, in particular, exemplify the qualitative approach in the formative evaluation of a web-based training program. Its results present agreement with previous findings from technical evaluation (See the poster Formative evaluation: using diverse tools in order to refine and improve instructional procedures in distance training). Such practice has shown great value, once the qualitative methods have added specific information that clarifies data collected with quantitative tools. Combining both qualitative and quantitative methods is suggested for further research with web-based training, once it allows a more holistic treatment of the multiple variables involved in the continuous learning process that should take place in a sustainable organization of work. DISCUSSION AND RECOMMENDATIONS Abbad, G., Gama, A. L. G. & Borges-Andrade, J. E. (2000). Treinamento: Análise do relacionamento da avaliação nos níveis de reação, aprendizagem e impacto no trabalho. Revista de Administração Contemporânea, 4, Borges-Andrade, J.E. (2002). Desenvolvimento de Medidas em Avaliação de Treinamento. Estudos de Psicologia, 7 (Número Especial), Kirkpatrick, D. L. (1976). Evaluation of Training. Em R. L. Craig (Org),Training and Development Handbook (pp ). New York,NY: Mc Graw-Hill. REFERENCES RESULTS University of Brasília - Brazil Institute of Psychology Impacto: Research on Training and Organizations of Work Authors: Lidia Parachin André Wogel Gardênia Abbad Maria Emília Araújo Talita Custódio Karen da Matta INTRODUCTION PRONEX Fubra Table 2: Factorial Loads for the items in Study 1 Study 1: Descriptives: Courses ‘MA’ and ‘FC’ were both well evaluated, as it is shown in the Table 1. Factorial analysis and internal consistency analysis presented two factors: Traditional procedures (alpha de Cronbach=0,97), and Procedures based on new information technology (alpha de Cronbrach=0,88), as it is shown in the Table 2. Study 2: First, researchers proceeded to a technical evaluation of the didactic material of the course, making use of the The Analysis of the Didactic Material Checklist. Then, participants of the course answered to a questionnaire containing five open questions about: 1. application of what participants learned; 2. what aspects of the participant’s work improved because of the training program; 3. participants’ opinion about the exercises; 4. what aspects should be improved in the training program; 5. what participants would like to learn in the course and haven’t learnt. Study 1Study 2 What was the context? 2 distance education courses: (1) Program ‘MA’ — 60h at distance, 40h at presence; 710 participants. (2) Program ‘FC’ — 60h at distance; 223 participants. A web-based training program with the main objective of capacitating consultants from a big Brazilian financial institution. What was the object? Training reaction to procedures: (a) Traditional procedures — the ones common to both distance and presential modalities (e.g.: quality of instructional objectives, sequence of content); (b) Procedures based on new information technology (e.g.: use of chat, tutoring). Training reaction to different aspects of the course, such as: (a) Instructional Strategies; (b) Content; (c) Layout; (d) Type of language used; (e) Tutoring. How was it done? Quantitative method: Questionnaire 12 items Likert scale (0=awful; 10=excellent) Qualitative methods: Qualitative Analysis of the didactic material; The content analysis of learners’ answers to open questions. Technical Evaluation - Weakness of exercises - Layout problems - Navigability problems - Lack of practical tools - Lack of interactive alternatives - Weakness of the content Participants' Answers Positive: “Today I feel more prepared to sign a contract with a client, acting accord his style, to negociate” / “The content helped me in the selection of the consultants in my area.” Negative: “I haven’t applied it yet, at any kind of consult”. / “The contents at the web training are theorics, distant of the practice and the day by day of a internal consult.” Positive: “I learned the differences of many types of a consult.” “We feel more secure to propose actions”. Negative: "Nothing. I already knew what was there." Positive: “The exercises were very good, because they helped me to memorize the contents” / “They are relevant.” Negative: “There were few exercises” / “In general, they were not good”. / “I exercised my memory, not the application of the theory.” / “Totally incompatible with the contents in the unit.” Navigability Interaction Grammar Errors Applicability Sequence and Distribution of content “The use of tools for projects.”/“The real role of a consultant at the organization.”/“Practical guidelines about leading with clients.” Table 1: Descriptives in Study 1 Table 3: Results in Study 2