1. What do the kids think? A quantitative analysis of feedback questionnaires in standardised reading tests Eva Konrad & Annabell Marinell

2. Outline Research questions Secondary school types in Austria The research instrument Results Conclusions A. Marinell & E. Konrad IATEFL TEASIG 2011

3. Research questions Is there a connection between – the test takers’ familiarity with the topics – their interest in the texts used – how difficult the test takers felt the texts to be – their familiarity with the test methods and how well they felt it tested their reading ability? Is there a connection between the test takers’ total scores and how well they felt the standardised reading test tested their reading ability? Is there a difference in test takers’ responses from different school types? A. Marinell & E. Konrad IATEFL TEASIG 2011

4. Secondary school types in Austria A. Marinell & E. Konrad IATEFL TEASIG 2011 AHSBHS 1.HUM: comprise business studies, theory in the respective area as well as compulsory work placements 2.HAK: secondary schools for occupations in the business sector 3.HTL: secondary schools for engineering 4.BAKIP: nursery schools 5.BASOP: colleges for social education 6.HLFS: colleges for agriculture and forestry 1.Classical secondary academic school 2.Secondary academic school emphasizing mathematics and science 3.Secondary academic school emphasizing economics 4.Upper-secondary academic school specializing in instrumental music 5.Upper-secondary academic school specializing in fine arts

5. Research instrument: test booklet 4 independent reading tasks Test formats: Multiple Choice (MCQ) Multiple Matching (MM) Sequencing (SEQ) Note Form (NF) True/False/Justification (TFJ) (dropped: True/False/Not given) Answer sheet Questionnaire A. Marinell & E. Konrad IATEFL TEASIG 2011

6. Research instrument: questionnaire A. Marinell & E. Konrad IATEFL TEASIG 2011 Questionnaire excerpt

7. Selected questions for research 1.How familiar were you with the topic of the texts? 2.How interesting did you find the texts? 3.How difficult did you find the texts? 4.How familiar were you with the test methods? 5.How well could you show your reading competency in this test? A. Marinell & E. Konrad IATEFL TEASIG 2011

8. Face validity “The extent to which a test meets the expectations of those involved in its use, e.g. administrators, teachers, candidates and test score users; the acceptability of a test to its stakeholders.” (McNamara 2000) For our research: How well did the test takers feel the standardised reading test was able to measure their reading competency? A. Marinell & E. Konrad IATEFL TEASIG 2011

9. Test population Trial 2008: AHSTrial 2011: AHSTrial 2011: BHS (N=1665)(N=3041)(N=914) School type Booklet 1184Booklet 1303Booklet 1 124 BAKIP Booklet 2192Booklet 2306Booklet 2 128 HAK Booklet 3186Booklet 3273Booklet 3 160 HAK Booklet 4184Booklet 4297Booklet 4 115 HTL Booklet 5183Booklet 5299Booklet 5 112 HTL Booklet 6181Booklet 6388Booklet 6 92 HUM Booklet 7168Booklet 7385Booklet 7 91 HUM Booklet 8182Booklet 8385Booklet 8 92 HUM Booklet 9117Booklet 9405 Booklet 1088 A. Marinell & E. Konrad IATEFL TEASIG 2011

10. Method Programme MSB statistics programme Developed by Gerhard Marinell and Gabriela Steckenberger A. Marinell & E. Konrad IATEFL TEASIG 2011 Method

11. MSB statistics interface

12. Data matrix A. Marinell & E. Konrad IATEFL TEASIG 2011 Significance level as quoted in Henning (1987): α=0.05

13. Question 1 Is there a connection between the test takers’ familiarity with the topics interest in the texts used how difficult they felt the texts to be and how well they felt their reading ability was tested? Is there a connection between these factors and the test takers’ total score? A. Marinell & E. Konrad IATEFL TEASIG 2011

14. Values in %

15. A. Marinell & E. Konrad IATEFL TEASIG 2011 Values in %

17. Question 2 Is there a connection between the test takers’ familiarity with the test methods and how well they felt their reading ability was being tested? A. Marinell & E. Konrad IATEFL TEASIG 2011

18. Values in %

19. Question 3 Is there a connection between the test takers‘ perception of how well their reading competence was tested and their total scores? A. Marinell & E. Konrad IATEFL TEASIG 2011

20. Values in %

21. Conclusions We could observe certain tendencies: – Correlations in all of the factors were more frequent in 2011 than in 2008 – Correlations with face validity were more frequent than with total score Further analysis of feedback questionnaires is needed We will be able to say more about BHS after more trials We still have several questions: – Are there differences between the different BHS school types? – What further information might we gain from the open questions? A. Marinell & E. Konrad IATEFL TEASIG 2011

22. Bibliography Alderson, C. J., Clapham, C., & Wall, D. (2005). Language test construction and evaluation. Cambridge: CUP. Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge: CUP. Henning, G. (1987). A guide to language testing: development, evaluation, research. Cambridge, Mass: Newbury House Publishers. McNamara, T. (2000). Language testing. Oxford: OUP.

23. Thank you for your attention and enjoy the rest of the conference! A. Marinell & E. Konrad IATEFL TEASIG 2011