1. ImpaCT 2 & the continuing quest to understand ICT & Attainment
MIICE Conference, November 04
Malcolm Hunt, Becta

2. Presentation Outline Key findings from ImpaCT 2
Post ImpaCT 2 work about ICT & Attainment
Context issues, including ICT Pedagogy
Summary & Conclusions

3. Key findings from ImpaCT 2

4. Pre ImpaCT 2 thinking RESEARCH ISSUES:
Difficult to isolate any single aspect such as ICT from other factors.
Factors are likely to be at school, organisational and pupil levels.
All pupils have access to ICT so control group not an option
Using a large enough sample for statistical analysis, for a representative study
Robust/ cumulative qualitative evidence as well to understand practice

5. ImpaCT 2 Overview
Longitudinal Study over 3 years between 1999 and 2002.
Explored the relationship between networked technologies and attainment
Sample of 2179 pupils from 60 schools: - 30 Primary Schools - 5 Special Schools & - 25 Secondary Schools in England

6. ImpaCT 2 Strands & Design
Measures of ICT Use: Levels of Use: from questionnaires Types of Use: from questionnaires, pupil/teacher logs & visits by researchers
Sample: 700 pupils at KS 2, 3 and 4
Strand 1: Considered the relationship between pupils’ use of ICT & their performance in National Tests & GCSEs as a measures of attainment.
Strand 2: Developed and applied a variety of methods to establish how pupils use ICT, in particular out of school.
Strand 3: Explored the nature of teaching and learning involving ICT in various settings
Methods used : - Pupil researcher logs - Questionnaire on internet use - Special report of a key learning event with ICT -Concept mapping task
15 Cases studies with a focus on the views of pupils, teachers and parents.

7. How did the researchers find out how ICT affected attainment?
700 pupils for each key stage
Value-added data compared for ‘high ICT’ and ‘low ICT’ pupils
13 comparisons – English, mathematics and science at KS2, 3 and 4 and – KS4 history, geography, MFL & D&T.

8. Establishing how ICT affected attainment
Positive relative gain
Pupil A
Expected score
Negative relative gain
Positive relative gain
Pupil B
Expected score
Negative relative gain
Baseline score
National Test/ GCSE score
In the illustration Pupil A has a higher test score than Pupil B, but performed worse than expected, and so has a negative relative gain score.
Pupil B, by contrast, performed better than expected and so has a positive relative gain score. Comparison of these baseline scores with the final grades of pupils in National Tests or GCSEs taken in the summer of 2001 allowed the calculation of an expected score for every pupil. The difference of his or her actual result from that ‘expected’ is taken as a measure of ‘relative gain’.
A relative gain score of zero indicates that a pupil’s National Test score was as predicted by the baseline test, a positive relative gain score indicated that the pupil had exceeded expectations, a negative one that the pupil had not achieved the result expected. The size of a pupil’s relative gain gives an indication of how that pupil performed compared to other pupils starting from the same baseline position.
For example, a pupil who achieved a result one standard deviation higher than the mean for the population of pupils starting from the same position would achieve a relative gain score of + one. (Standard deviation is a statistical term for the average difference from the mean for a group of results).
Because the schools in the sample vary in their catchments areas and hence in the opportunities afforded by the home and neighbourhood, and because similar considerations apply to pupils within schools, it would have been inappropriate to accept raw National Tests and GCSE scores as criteria of final performance. There is large body of evidence indicating that relative gain can be calculated to measure the progress of individuals and groups over a period by adjusting final raw scores to take into account the score that might be ‘expected’ on the basis of initial achievement on one or more relevant tests. Such calculation goes a long way to producing a level playing-field for comparison.
Having established the relative frequency and context of use in each subject, the study explored the relationships between the use of ICT and performance in National Tests and GCSEs. These relationships were explored by comparing each pupil’s actual achievement with his or her predicted achievement, derived from ‘baseline’ scores of proven reliability and validity provided by Durham University. These scores were calculated from tests that the pupils had undergone approximately eighteen months earlier (during ). This comparison produced a relative gain score for each pupil, which is zero if the pupil did as predicted, positive if the pupil did better than expected, and negative if worse.
Pupil A has a higher test score than Pupil B, but performed worse than expected, and so has a negative relative gain score.
Pupil B, by contrast, performed better than expected and so has a positive relative gain score.

9. Key Findings from Strand 1
In 12 out of the 13 comparisons, ‘high ICT’ pupils achieved on average higher value added than ‘low ICT’ pupils.
Of the 12 comparisons 4 were statistically significant: - English at KS 2 - Science at KS 3 - Science at GCSE & D&T at GCSE
In some cases the difference in average performance was equivalent to 0.2 of a National Curriculum level at KS 2/3 and 0.5 of a GCSE grade at KS4.
One level is thought to relate to around 2 years in a pupil’s development – ie they are expected to progress by the order of 0.5 of a level a year
One level relates to around 2 years in a pupil’s development – ie they are expected to progress by the order of 0.5 of a level a year

10. ImpaCT 2- Getting behind the data
Researcher judged the amount of ICT use overall to be low.
The number of networked computers rose dramatically during the period from
Types of ICT use as well as the amount of ICT use play a part in terms of impacts on attainment/learning
A range of other context factors play a part
The researchers found that during the period from , the number of networked computers rose dramatically within all schools. Teachers found it challenging to integrate networked technologies within their lessons for a number of reasons including.
Teacher confidence and capability with ICT
Concerns about Internet safety
Problems in development of robust networks in school
Problems integrating legacy kit with new equipment
Access to these ICT resources
Lack of time for reflective practice and helpful guidance on teaching with ICT
“I’ve done all this NOF training, I’ve really got to grips with [ICT], its in my planning, the children are all fired up, but I can’t come in my class every week and think – is it going to work?” ICT co-ordinator Primary School
Many of these issues are now being addressed but at the time the data was gathered the 60 schools were at very different stages of developing their ICT capability.
However these same issues are likely to continue to be relevant for all but the most advanced schools in the country.

11. Getting Behind the Data English KS 2 - School by school comparisons
Exceptionally good teaching with less use of ICT?
The use of a particular intervention other than ICT?
If high use of ICT is related to improvements in English then you would expect a ‘staircase’ pattern of bars moving from left to right. Those schools with low ICT use would display negative gain scores and as you move to the centre the bars would move through 0 to register positive gain scores rising to the highest ICT using schools displaying the largest relative gain scores.
Schools in the study were grouped in descending order of their over all level of use of ICT. The highest user of ICT being school A, the lowest user of ICT school X. The position of any given school could vary from subject to subject to subject. In this case school K is the third highest user of ICT in English after school A and B and ahead of C and D.
The graph illustrates how even in the case of KS 2 English where there was a significant finding linking the level of ICT use to performance in KS 2 English test scores, you do not get a perfect staircase pattern across the graph. School R has a moderate level of ICT use in English but the English scores are greater than the higher ICT using schools such as A through to H
High
Low
Increasing ICT use
Mean relative gain for schools in order of ICT usage for English

12. Getting Behind the Data English KS 2: School by school comparisons
School T uses ICT more often than might be expected in English from the overall level of ICT use, yet English attainment is below the average
Maybe the totality of a pupil’s experience influences achievement, rather then an isolated use in a particular subject
Similar level of ICT use, very different relative gain score
High
Low
Increasing ICT use
Mean relative gain for schools in order of ICT usage for English

13. KS 2 English teachers identified:
The motivating effect of ICT was a common factor: - Captivated pupils who previously found it hard to engage - Positive shift in attitude involvement in learning activities
Types of use included word processing, publishing packages, use of the internet for research, software to aid spelling
Higher quality outputs encouraged greater commitment to writing tasks
Relevant software making the learning of key skills more fun (eg learning spellings)
Increased time for reflection/ refining the content of written work

14. Type & nature of the Research Questions
Maddux model of Stages in ICT research
Need to move on from whether ICT has general educational benefit to how and why ICT works
Need to refine our research questions and consider subsets of ICT - By subject - By ICT application eg , use of spreadsheets
Focus more on the context of use
STAGE 1: Computers produce general educational benefits
STAGE 2: Exposure to a specific application produces a specific educational gain
The Maddux model encourages us to ask more searching and refined questions about the relationship between the use of ICT and the impacts on learning and attainment in particular. Only by refining our research questions in this way can we move from questions about ‘whether’ ICT has an impact to the ‘how’ and ‘why’ questions.
STAGE 3: Which and how learners interact with teaching variables when using specific aspects of ICT

15. Asking the ‘how’ & ‘why’ questions about ICT & Attainment

16. ICT & Attainment Literature Review (Cox et al 2003)
Positive effects of specific uses of ICT on pupil attainment in almost all NC subjects
Most substantial in Maths, Science and English, where more uses being made of different resources in these subjects
In Maths and Science learning is most enhanced when the type of ICT use and the measures are closely related to specific skills and concepts.

17. ‘Deeper’ uses of ICT (Cox et al 2003)
Specific Uses
Subject
Impact
English
Science
Programming
and modelling
Maths
Word processing
and presentations
Simulations
Understanding and
applying concepts
Scientific reasoning
and explanations
Creativity and
developing meaning
Geography
Concepts and
relationships
History
Multimedia
resources
historical reasoning
MFL
Simulations and
role playing
Specific language
skills, confidence
Literature reviews not only reflect significantly more literature in the core subjects of English, Maths and Science, the pattern can also be seen in the levels of ICT use reported in surveys of teachers’ use of ICT across the curriculum, in English schools, especially in primary schools.

18. ICT & Attainment Literature Review
Impact appears strongest for ICT resources which have been embedded in teachers’ practices for a long time.
Emerging body of evidence about specific ICT communications eg , WWW, but the effects on attainment is not as consistent or extensive.
The crucial component is the teacher and their pedagogical approaches

19. Factors in ICT Pedagogy & Reasoning (Cox et al 2003)
Confidence in using ICT
Access to ICT Resources
ICT Pedagogy and Reasoning
Organisation & classroom practice
Teachers subject knowledge
Teachers pedagogical knowledge, beliefs & values
Planning & preparation beyond the classroom

20. Factors that affect teacher uptake & use of ICT (Mumtaz 2000)
Institutional: eg: Time, technical support, ICT resourcing
Resources: eg: Access to computers in the classroom Range of software available Lack of training and the quality of training
Teacher factors: eg: ICT experience, confidence/competence, Beliefs & theories about subject teaching, Knowledge & perceptions about the benefits of ICT Managing classroom activities/control Personal feelings towards ICT Fear of change
Main Barriers
ICT Confidence/ competence
Access to ICT
Training
Time
Technical problems/support
Main barriers, listed roughly in order of importance given to them by literature.
Teacher confidence affected by many other factors – will look at later
Access can be split into different types of problem – will explain in a moment
Training – many references to failings of NOF, most of literature suggests balance of skills and pedagogy.
Time – for training, exploring software, and preparing resources.
Technical problems – fear of things going wrong (links to confidence) and lack of technical support in schools.
Resistance to change – complicated issue, usually as a result of other factors. (Dawes)
Perception of benefits – teachers don’t always see how it will aid them, only see the time it takes them to do anything with ICT.

21. Broad agreement with the literature survey
Based on 172 practitioners surveyed at BETT 03 and the Education Show
Broad agreement with the literature survey

22. LACK OF TEACHER CONFIDENCE
Direction of arrows represent “could be caused by…”
Red arrows show a two way effect.
Lack of tech. support
Access at
school
home
Lack of personal access
Technical Problems
LACK OF TEACHER CONFIDENCE
Fear of things going wrong
Lack of Time
Lack of teacher competence
Lack of exploration
Personal access – at school and at home, so teachers can’t practise and build up confidence
Arrow direction indicates “could be caused by”, although not necessarily proven links. Red arrows show two-way effects.
Technical problems caused by lack of technical support.
Teachers frequently report a fear of things going wrong, caused by technical problems and lack of technical support.
Fear of things going wrong also directly related to lack of confidence.
Lack of competence caused by lack of pedagogical and also ICT skills training. Courses need to have both.
Confidence caused by lack of time – lack of time for exploration.
Lack of exploration also caused by lack access.
Lack of access at school caused by lack of technical support.
Fear of things going wrong caused by lack of ICT skills training.
Black arrows show other interrelations
Lack of ICT Skills Training
Lack of Pedagogical Training
Becta ICT Barriers report

23. Personal ICT Enablers
Personal access to ICT to allow the development of confidence and competence
Commitment and support from departmental and senior management
Involving whole staff in ICT decision making
Importance and involvement of a support community of peers
Perceived usefulness of ICT and ease of access and use
Becta ICT Enablers report

24. ICT Standards report-
Studied results of OFSTED inspections & QCA data on performance in national tests
‘ICT enablers’ that contribute to good ICT learning opportunities were: - ICT resourcing - ICT leadership & management - ICT teaching - General School Leadership - General Quality of Teaching
Becta’s research on ICT and Standards
Is there a ‘threshold’ effect?

25. ICT enablers and ICT learning opportunities (primary)%

26. ICT enablers and ICT learning opportunities (secondary)%

27. A developing model: ICT enablers
Improved Outcomes
The ‘building blocks’ of enablers that can help a school to make effective use of ICT.
ICT used effectively in classrooms for learning
Improved learning
Increased attainment
ICT deployed appropriately
ICT learning opportunities
NCSL/Becta SLICT courses
Leadership
ICT leadership
ICT resources
General teaching
ICT teaching
The ICT in schools programme
CfT and LfT
NOF Training, HOS initiative, CPD support materials

28. Stages in networked ICT implementation
ImpaCT 2: Strand 2 research: Somekh et al 2002
Pupils’ & Teachers’ Perceptions of ICT in the Home
TIMESCALE?
Stage 1: Provision of Equipment, infrastructure and support
Stage 2: Teaching ICT skills, often in specialist ICT lessons,
Stage 3: Integration of ICT into the curriculum subjects, including literacy and numeracy
During data collection, schools moved from Stage 1 to 2
Only a few teachers in a few schools began to move to Stage 3

29. Home / School Issues
The importance of ICT use in the home and for informal learning is increasing (ImpaCT 2 figs) - Primary School pupils: on average 3 hours a week compared to 1 hour at school - Secondary School pupils: on average 10 hours a week compared to 2.5 hours at school (4x)
Home ownership and internet access is increasing yet over a quarter of pupils did not have internet access at home. Potential disadvantage?
Pupils have an extensive awareness of the role of computers in today’s world (Type of equipment, use & purpose)
ImpaCT 2 Findings Teachers recommended website for use at home & in support of subject specific learning
ImpaCT 2 found that teachers were recommending web sites to children for use at home, and that there was a significant use of ICT and the internet for subject specific purposes, leading to an extension of learning in school to the home environment, and vice versa. This shows the power of ICT for establishing closer home-school links, which may well be of interest to practitioners
Public access to ICT in public libraries & internet cafés can lack flexibility & be expensive How might schools address this?
Students of all ages may develop ICT skills more quickly and easily than schools anticipate

30. Emerging Findings from Becta’s Research about ICT & Attainment
Growing body of evidence of an ICT effect eg ImpaCT 2 study, SANDS Report
Beginning to appreciating the use of ICT in specific subjects- Subject ICT traditions are important eg ICT & Attainment literature reviews
Identifying important elements in the effective embedding of ICT- Institutional conditions are critical eg Barriers and Enablers research, ICT and Pedagogy research

31. Conclusions
ICT and Standards: ensuring that we can go beyond the research questions as to ‘whether’ ICT has an impact, to ‘how’ and ‘why’ it does.
ICT pedagogy: ‘moving on from the ‘foothills of pedagogy’’ - better understanding ICT use in different subjects and with different technologies - better understanding mixed approaches blending ICT & traditional methods, including Personalised Learning
The embedding of ICT - understanding teacher engagement, CPD needs - for Institutional Development & the E Confident school

32. Contact details Becta Millburn Hill Road Science Park
Coventry CV4 7JJ
Becta Research web site:
Malcolm Hunt: Assistant Director, Evidence and Evaluation,

33. ImpaCT 2 & the continuing quest to understand ICT & Attainment
MIICE Conference, November 04
Malcolm Hunt, Becta

34. Sources: ImpaCT 2: http://www. becta. org. uk/research/research. cfm
Sources: ImpaCT 2: ICT and Attainment Report: ICT and School Standards: Barriers & Enablers Reports: