Using Diagnostic Assessment Results to Inform Development of Academic Support Interventions Presentation by Dr. N. Phewa at NACADA Conference, 2013 (Maastricht)

INTRODUCTION The study aimed to determine students’ academic needs by administering an on-line diagnostic assessment tool; The classic tool assessed 13 skills, 7 skills clusters and 3 competencies; On completion of the tool students received immediate feedback – an overall achievement score Two weeks later an analysis of skills results was made available to students and lecturers – highlighting performance per cluster and competency

METHODOLOGY A classical test was developed by a team relevant stakeholder departments; It was piloted with 156 second semester second level Management Accounting students; Results were analysed with Classical Item Analysis computer program to determine the tool’s performance; Students’ scores were analysed according to the assessed skills clusters and competencies on an excel spreadsheet

METHODOLOGY The tool was redeveloped to produce a second version; The second version was administered to 422 first semester second level Management Accounting students; The same analysis procedures as were carried out on the first pilot were conducted; and A third version of the tool resulted; This version was administered to a mixed second semester first and second level cohort comprising 485 students

SKILLS CLUSTERS ASSESSED Lexical items; Sentence construction; Understanding communicative function/purpose; Reading for meaning; Appropriacy to context; Understanding information presented visually; and Understanding basic numerical concepts

COMPETENCIES ASSESSED Linguistic; Discourse; and Visual and Numerical

FINDINGS Our findings largely relate to bottom performers on the tool (those scoring below 50%) Skills clusters assessed % scoring below 50%

FINDINGS Competencies assessedLinguisticDiscourseVisual & Numerical % of cohort scoring below 50%10819

FINDINGS PERFORMANCE IN SKILLS CLUSTERSPERFORMANCE IN COMPETENCIESTOTAL LinguisticDiscourseQuantitativeTotal

THE WEAKNESSES OF THIS STUDY A voluntary pilot ‘low-stakes’ assessment; Relies on sub-scoring to produce results per skills cluster and /or competency; Sub-scoring has been found to temper with test validity (Skorupski, 2008) especially when no augmentation procedures have been carried out, as was the case with the current study.

MITIGATION AGAINST THE WEAKNESSES It is proposed that the test be made a high-stakes assessment tool with consequences; Skills are grouped together according to alliance to form clusters; At least 10 items assess each cluster; It is proposed that a predictive validity study be carried out considering all variables that may be influential on students’ performance; and Reliability augmentation procedures be carried out to increase the reliability of the sub-scores, which in turn will increase the validity thereof.

THE STRENGTHS OF THIS STUDY Provides students strengths and areas needing further academic development per skills cluster assessed; Enables provision of advice to faculty and mentors relating to academic needs to focus on in intervention programmes per student/ cohort; and Enables development of focussed intervention per student category.

CONCLUSION The same kind of results were obtained for all the other pilots, i.e. achievement results do not tell us what the students could not do in a given examination; Even some academically prepared students may be experiencing some areas of academic development, which achievement results do not expose; These findings highlight the need for diagnostic assessments in helping not only the overtly ‘unready’ students, but also the ‘seemingly’ ready ones, as illustrated in the previous slide. Therefore, it would seem that considering the results of such diagnostic assessments would be sensible when developing academic support interventions.

(012)