2. Seminar Overview Welcomes, Introductions Background to e-asTTle
How e-asTTle works - rules of engagement
Thinking differently about testing – Item Response Theory
Misconceptions
Creating a test
Customised and Adaptive Tests
Copy tests and Similar tests
Writing test overview
Reports & next steps
Interpreting the reports
e-asTTle updates
Administration of e-asTTle
Reflection and Evaluation

3. Burning issues….
Write burning issues on whiteboard

4. Electronic assessment tool for teaching and learning
INTRO:
e-asTTle stands for…

5. Basic Rules of Engagement
The data will be used to inform teaching and learning
Students will share in the analysis of the data
Testing will be by current achievement levels, not age
The results will be interpreted in conjunction with other forms of assessment
The reports are only the beginning of analysis – not the analysis itself
The student’s test score in NOT an Overall Teacher Judgment

6. Think differently about testing – changing our hardwired thinking

7. Think differently about testing
Students A, B and C all sit the same test
The test has ten items
All items are dichotomous (score 0 or 1)
All three students score 6 out of 10
What can we say about the ability
of these three students?

8. Think differently about testing
Item 1 2 3 4 5 6 7 8 9 10
Score
e-asTTle
Difficulty
Bob 
Emma
Harry

9. Think differently about testing
Item 1 2 3 4 5 6 7 8 9 10
Score
e-asTTle
Difficulty
Bob 
1342
Emma
1392
Harry
1435

10. Think differently about testing
Item 1 2 3 4 5 6 7 8 9 10
Score
e-asTTle
Sublevel
Difficulty -3 -2 -1
Bob 
1342 2A
Emma
1392 3B
Harry
1435 3P

11. Think differently about testing
Two students get 6/10 correct.
However, this does not mean they will receive the same e-asTTle scale score.
Item Response Theory is based on the difficulty of items a student gets correct - not how many.
Implication
Raw scores mean very little in e-asTTle.
Get participants to look at cartoon. Read page 2. Check in.

12. Misconception #1 “Everyone needs to do the same test for a fair comparison”
If students of wide ability take the same test it is really only the ‘average’ sector where any useful information will be gained.
Getting too many right or wrong provides us with very little information.
Implication
Test students on current achievement levels, not age.
It may be necessary to create up to four different tests for some cohorts. Each student is then assigned to a test which is most suitable for him/her.

13. Misconception #2 “100% is best?”
If we want to use asTTle to inform us, the students need to get some answers wrong.
A student that gets 100% correct gives little information on where his/her next steps in learning are.
Implication
We must make the tests challenging, even difficult for the students.

15. Misconception #3 “Everyone must take the test at the same time in the same place”
E-asTTle provides the opportunity/challenge to test students at different times and places, especially when taking an online test.
Students, if allowed in the assignment of the test, can even sit a test at home.
The MAIN PURPOSE of the test will dictate when, where and how testing can take place.

16. Misconception #4 “e-asTTle means you take the test on the computer”
In many cases computer based testing is the best way forward.
However, paper based testing should also be considered depending on your situation.
Paper based testing is the recommended method for areas such as Geometry or Measurement where interaction with graphics is required.

17. CHECKPOINT 1 What were the key messages for me?
What action do I need to take?

18. Creating Tests The following tests can be created: A Customised Test
An Onscreen Adaptive Test
A Writing Prompt
And the following actions can also occur:
Copy an Existing Test
Create a Similar test
Add Numeracy Data
Page 7. Then look at test duration/ strands etc pages 3-6

19. Customised & Online Adaptive Tests
Customised Test
This test is customized by the user and the process allows the person creating the test to select from a range of curriculum strands, different processes or time and control features, such as on-line versus paper and pen.
Online Adaptive Test
The main difference between an adaptive and a customized test is that e-asTTle adjusts the difficulty of the test questions while the student is sitting the test based upon the responses they are providing.
Page 7. Then look at test duration/ strands etc pages 3-6

20. Adaptive test
Page 7. Then look at test duration/ strands etc pages 3-6

21. Creating a Writing Prompt
School / teacher chooses a purpose and a prompt
Student write for up to 40 minutes, maximum
Writing is scored using…
rubric
annotated exemplars
structure and language notes
Curriculum level is not part of the marking.
When rubric scores are entered a curriculum level for each element and an overall score will be available
SECTION TWO – ENSURING HIGH QUALITY DATA
Test creation – Writing
This is the process for a writing test…
School / teacher chooses a purpose and a prompt
When choosing a purpose think about the complexity of the structure/purpose e.g. narrate and persuade is more complex than recount or describe
When choosing a prompt, think about the level of abstraction e.g. explain the need for a new community facility
Some prompts are written particularly for younger students
E.g. describe and recount, e.g. The Girl
Student write for up to 40 minutes, maximum
Writing is scored using…
rubric
annotated exemplars
structure and language notes
Curriculum level is not part of the marking.
When rubric scores are entered a curriculum level for each element and an overall score will be available
You may want to consider different prompts for different groups.

22. Creating a similar, or copying a test. Why would you?
Creating a similar test means that you can create a new test using the same settings as a previous test but it will result in a new set of questions
Copying a test allows you to use the same test with a different group of students, or, with the same group of students but at a different time. It will have the same questions. The benefit is that it allows you to keep the two reports separated.
If you use the same test twice the data will be mixed
Copy test
makes the same test, but will keep the reports and data separate
Similar test
same settings as a previous test but with new questions
If you give kids a copy/similar test it will have the same settings
e.g. L2 = 80% and L3 = 20%
So the test will only allow them to go so far
INSTEAD Take the second test up a notch
e.g. L2 = 60% and L3 = 40%

23. CHECKPOINT 2 What were the key messages for me?
What action do I need to take?

24. Time to have a go… e-asTTle on TKI
e-asTTle Training site – Teacher login
e-asTTle Training site – Student login
Page 7. Then look at test duration/ strands etc pages 3-6

25. CHECKPOINT 3 What were the key messages for me?
What action do I need to take?

26. Online tests – Marking open response questions
Closed questions (i.e. multi choice or true/false type questions) do not require any data input or marking. The programme marks them automatically
Open questions are marked by question with student responses clustered.

27. Entering Paper Tests data

28. Entering e-asTTle Writing scores

29. e-asTTle Reports

31. The Console Reports

32. The Console Reports
These are for analysis for your report. They are not a reporting device within themselves.

33. The Comparisons Console Report (Schools like mine)
Comparison reports – Good comparative data but are testing like with like

34. The Multi-test Console Report
Multi test report eliminates the concern that you can’t see overall performance of the group if they take different tests.

35. The Individual Learning Pathway Report

36. The Individual Learning Pathway Report

37. The Individual Learning Pathway Report
Standard error of measurement around the students score (± )
This is like the 'margin of error' reported in political polls
Two out of three times the student’s true score will lie somewhere between the top and the bottom of the red circle
A difference in scores needs to be >22 to be ‘statistically significant’

38. Error of Measurement
Error of Measurement is always higher at the extremes of a scale than in the middle of a scale… no matter which tool you use!
Individual error of measurement in writing, maths and reading.
e-asTTle standard error of measurement in reading and maths is 22 points and in writing it is 80 points .
SECTION THREE– ANALYSIS FOR TEACHING AND LEARNING
Error of measurement
Error of Measurement is always higher at the extremes of a scale than in the middle of a scale… no matter which tool you use!
Individual error of measurement in writing; and now in maths and reading too
e-asTTle standard error of measurement is 15 points (22 points)
1971 ± 119 (>6B) Range is 1852 (6B) to 2090 (>6B)
1488 ± 45 (2A) Range is 1443 (2P) to 1533 (3P)
892 ± 121 (1B) Range is 771 (1B) to 1013 (1B)

39. Error of Measurement – use cut scores
2A B
+ 6
2P P
+ 47
SECTION THREE– ANALYSIS FOR TEACHING AND LEARNING
Error of measurement
Use sublevels and scores, not just sublevels
A learner can have accelerated progress and stay within the same sublevel
A learner can shift sublevels but only based on a few points
1st example
Not many points shift
Within the error measurement, less than 22, so not statistically significant
But went up a sub level
Even into a new level
= don’t celebrate things that aren't worth celebrating
2nd example
Stayed in same sublevel, 2P
But big shift of 47 points
= could miss this shift and it should be celebrated
Use cut scores when analysing shifts
Writing cut scores separate (next slide)

40. The Individual Learning Pathway Report - Criterion information
When a dash “ – “appears in the e-asTTle ILP it means that the student has failed to get more than 3 questions correct in that strand.
If a dash appears in the overall score it doesn’t mean the student only got 2 correct answers in the test!

41. The Individual Learning Pathway Report - Interpreting the quadrants
Unexpectedly correct
Harder than student’s overall ability
Strengths
Unexpectedly wrong
Easier than or equal to the student’s overall ability
Gaps
Correct as expected
Achieved
Wrong as expected
To Be Achieved
Correct
Easy Items for this student
Incorrect
Hard Items for this student

42. e-asTTle reports – checking understanding…
Freddie Fangdangle scores 3B in reading or maths
Questions harder than 3B which Freddie got correct
Questions harder than 3B that Freddie got wrong
SECTION THREE– ANALYSIS FOR TEACHING AND LEARNING
The quadrants – checking understanding
Let’s check that we have shared understandings of the reports…
This is a Individual Learning Pathway for Reading or maths
It has 4 quadrants
Freddie got a 3B overall
Strengths…
Questions harder than 3B which Freddie got correct
Achieved…
Questions equal or easier than 3B which Freddie got correct
To be achieved…
Questions harder than 3B which Freddie got wrong
Gaps…
Questions equal or easier than 3B which Freddie got wrong
THEY ARE STRENGTHS AND GAPS COMPARED TO THAT INDIVIDUALS OVERALL LEVEL, NOT COMPARED TO THEIR AGE OR YEAR LEVEL!!!
Questions equal or easier than 3B that Freddie got correct
Questions equal or easier than 3B that Freddie got wrong

43. e-asTTle reports – checking understanding…
Freddie Fangdangle scores 3B in writing
Element scores more than a sublevel above 3B
e.g. 3A or higher
SECTION THREE– ANALYSIS FOR TEACHING AND LEARNING
The quadrants – checking understanding
This is a Individual Learning Pathway for writing
It has 3 of the 4 quadrants
Freddie got a 3B overall in writing
Strengths…
Any elements more than a sublevel above his overall level
3A or higher
Achieved…
Any elements within a sublevel of his overall level
2A, 3B or 3P
Gaps
Any elements more than a sublevel below his overall level
2P or loweR
THEY ARE STRENGTHS AND GAPS COMPARED TO THAT INDIVIDUALS OVERALL LEVEL, NOT COMPARED TO THEIR AGE OR YEAR LEVEL!!!
Element scores within a sublevel of 3B
e.g. 2A, 3B or 3P
Element scores more than a sublevel below 3B
e.g. 2P or lower

44. The Individual Learning Pathway Report - Interpreting the quadrants

45. The Individual Question Analysis report

46. Individual Question Analysis – Student Speak

47. The Group Learning Pathway Report

48. The Group Learning Pathway Report

49. The Group Learning Pathway Report

50. The Curriculum Level Report (aka Skyline)

51. The Curriculum Level Report (aka Skyline)

52. The Curriculum Level Report (aka Skyline)

53. The Curriculum Level Report (aka Skyline)

54. The What Next Profile

55. The What Next Profile – in e-asTTle online

56. The What Next Profile – on the TKI site

57. The What Next Profile

58. The Tabular Report

59. The Tabular Report – Excel file (.csv)

60. The Progress Reports

61. The Progress Report – for 1 student

62. The Progress Report – for a group

63. Manage Students Area

64. Target setting

65. Student Result Summary
Have a look at some of your own students…
SECTION THREE– ANALYSIS FOR TEACHING AND LEARNING
Student Result Summary
Shows results from test at previous schools
Uses the NSN number
Through manage students, not reports
See the Student Result Summary help sheet in pack
Spend some time looking at your students now

66. Manage students – Student details

67. Manage students – Student details
Download Student Logins
by PDF or csv (excel)
As a PDF you can print out as a card, hand it to the student and receive back once the password has changed.
To reset password go into “Manage existing students”, choose students and then click “reset password”.

68. Manage students – Using groups

69. Manage students – Using groups
Some groups will automatically be created when SMS data is imported. Most often Class, Year etc.
However you can create custom groups that enable the viewing of reports etc. for that particular group
By doing this you DO NOT need to test a certain group again. You can use existing data but just view reports for the selected students in your group
Examples of custom groups could be:
Your top reading group
An extension or remedial group
Maori boys or similar groups that need to be monitored
Target students

70. e-asTTle updates SECTION FOUR– FINAL THOUGHTS
Regular upgrades to e-asTTle
This is done regularly
Usually at night or in the holidays
When you log into e-asTTle it will let you know if a shut down for an upgrade is coming up
On the TKI e-asTTle home page there is usually some info to let you know what they did
There focus for the next few months is changing the online testing so that it is easier for students, less problematic and able to be done on ipads
If you are interested in helping with this, let us know

71. CHECKPOINT 4 What were the key messages for me?
What action do I need to take?

72. Can only be set up by someone with an e-asTTle School Leader role.
External Coordinators
External Coordinators are people who may not be staff members, but wish to access school data for various reasons.
Can only be set up by someone with an e-asTTle School Leader role.
SECTION FOUR– FINAL THOUGHTS
External Coordinators
External Coordinators have a special role in e-asTTle.
They can gain access to multiple schools and aggregate and report data across these schools.
ECs are people who may not be staff members, but wish to access school data for various reasons.
Only those with a School leader log in can do it
Click Maintenance on the left hand menu
There are several different EC roles:
PLD Providers – Access to e-asTTle data may help a PLD provider to develop effective teaching and learning programmes. They may wish to view e-asTTle data to prepare themselves for workshops, view student reports to track progress and carry out various other activities within the school they are contracted to.
Cluster Coordinators – Schools may belong to a cluster as part of a Ministry initiative or professional learning community. It is the duty of a cluster coordinator to track and report on cluster progress. They may wish to create a collection of common tests for groups across cluster schools and report across the cluster.
Researchers – A researcher appointed by the Ministry or a university, may wish to collect specific data or monitor general progress or achievement for a specific purpose. Researchers may need to create collections of common tests, track and report on student progress across multiple schools.
Temporary External Coordinators – There are circumstances where it may be useful to gain temporary access to data from multiple schools. For example, a Year 9 Dean may wish to create an entry test for year 8 students to sit before they enter high school the following year.

73. Future seminars…
Dealing with data seminar - An introduction to using Excel in schools
This practical seminar is for school leaders who need support in learning to use Excel when working with student achievement data.