BiH Test Piloting Mary Jo DI BIASE
In the beginning……
In the beginning…… Workshops on: item development validation practice in oral interviews rating In the beginning……
Bosnia’s Items ready Italy’s guinea pigs ready
Summary Test population background Trialling phases Statistical procedures Results …. and more
Test population background Staff officer’s career Course specifications JFLT –correlation
Staff officer’s career ISSMI: selected Army officers one year duration including language course with final SLP 3333
Test population background Three-month course Staff officer’s career 34 hours a week Course specifications Additional activities JFLT –concurrent validity Periodic diagnostic tests
Test population background Staff officer’s career Piloting held two weeks before JFLT Both are proficiency tests Course specifications Both based on STANAG 6001 ed. 2 JFLT –correlation Both have similar test types
Summary Test population background Trialling phases Statistical procedures Results …. and more
Listening & Reading Items in booklet form: total 60 level two and three items Instructions and proctoring Questionnaires
Writing Speaking 15 minute interview between interviewer and candidate (observer in background with rating scale) Writing Writing scripts for inter rater reliability Trialling prompts to check for level appropriateness, etc
Summary Test population background Trialling phases Statistical procedures Results …. and more
Score Distribution Cluster: mean, mode, median Dispersion: standard deviation, range
Item Behavior Classical Item analyses (Facility Value &Discrimination Index) Distracter Analysis
Reliability Scale if item is deleted Inter-rater reliability coefficient (Speaking and Writing)
Interpretation of Results Items in relation to: Facility Value and Discrimination Index Distracter behavior (ambiguity, implausible options, etc) Overall mean of test Reliability of test if item is deleted
Example According to the report… From an economic report (common knowledge-discarded) According to the report… approvals of bank loans have gone up recently. the UK economy is severely going downhill. recession started earlier than it was thought. positive figures are predicted for the next year.
C lassical I tem A nalysis Key B 94% A 0% C 5% D 1% FV 94% DI 0,05
Summary Test population background Trialling phases Statistical procedures Results …. and more
READING LISTENING Mean 60,87% Median 60,40% Mode 67,79% Range 62 Standard Deviation 13,35% Skewness -0,486 Kurtosis 0,043 READING Mean 77,05% Median 80,00% Mode 75,00% Standard Deviation 13,33% Range 60 Skewness -1,150 Kurtosis 1,477
Listening Reading
Reliability
Speaking Inter-rater reliability between Interviewer (holistic rating) observer (analytic rating) trainer in background Language generated by prompts (wording, level alignment) Tester conduct and elicitation techniques during interview
Correlation Coefficient: Writing Inter-rater Reliability Correlation Coefficient: 0,617 0,517 0,458 0,250
WRAPPING UP Overall beneficial: positive statistical results (30% items discarded) - additional ‘live’ training - more piloting needed
Summary Test population background Trialling phases Statistical procedures Results …. and more
First Official Administration of BiH test:
Fotografie Say…..
The BiH Testing Team in Rome
Thank you!!! maryjo.dibiase@unipg.it maryjo.dibiase@gmail.com