1. Educational Communication & E-learning
Cappelletti Ingrid
Matteo Barbero
Barbara Bragatto
Szymon Janikowski
Paweł Gilewski

2. Summary Sustainability Evaluation Cost & Benefit of Forum
Evaluation definition
Measure
RACE
Type of test
Objective test
Subjective test
Donald Kirkpatrick’s learning evaluation theory

3. Sustainability
We will be using an example of
FORUM
Number of posts – wrong criterion of measuring effort of a forum user. Rule „The more, the better” doesn’t work here.
For learning purposes it’s necessary to introduce
rules
time constraints
teacher’s and students’ effort. 3

4. Sustainability – example 1
Topic: Distance Education in IT Universities (50 students)
Rule
Time constraint
Student’s time consum.
Teacher’s time consumption
Post one very thoughtful message
By date x
1 hour to write
50 x 2 min =
100 min
Read 3 messages of your colleagues and give them feedback
Date x + 1 => date y
30 min to read 15
30 min to write replies
150 x 0.5 min =
75 min
Read the comments you have got and reply to them in a single wrap-up message
Date y + 1
=> date z
30 min to prepare wrap
50 x 1 min =
50 min
SUM
2.5 h
3h 45 min

5. Sustainability – example 2
Collaborative reading
of book with 8 chapters (153 pages)
as a part of 84 h course
~ 15 hours - too much!
Solution:
Everyone should choose 1 chapter and summarize it.
Reading: 2 hours Writing: 40 minutes
Everyone reads ALL summaries, then focus on two of them and comment them
Reading: 30 minutes Comment: 10 minutes
Remember about time constraints – deadline with time-zone
Not enough time to demand that every student reads th whole book.

6. Evaluation definition
learning process
ability level L
Measuring profit is testing 20
profit L 5 t1 t2
time
What you can tell about the
results of the test is called
EVALUATION
CMS
Communi-
cation
Class
manag.
Evaluation
& Testing

7. Measure styles In/Out or Pass/Fail Cardinal measure
E.g. driving license tests
Cardinal measure
E.g. chess and tennis players ranks
E.g. eligible & not eligible lists
Shows ranks, not distances between examinees

8. Measure styles Scale points, grades Most common school grading systems
Not all areas of scale covered
A mark is often not linearly dependant on one’s knowledge
Hidden psychological issues: 5++ or 6--
Paolo
Francesca
min
max
points

9. RACE An acronym for criteria a test has to meet. It must be:
Relevant and measure what was intended
Acceptable by the examinees, fair and appropriate
Comparable - test results may be used to examine and note the similarities or differences among the tested
Economic as correct strategies and amounts of resources are assigned for examining

10. Evaluation: Type of test
Objective test
Subjective test
Definition:
Test where you can not assign a score to advance require the presence of an expert
Example:
short essay
oral text
Advantages:
You can assess additional aspects in addition to knowledge as the ability to present arguments in a clear and decisive way.
Definition:
Test where you can assign in advance a score to every possible output
Example:
questions with answers true false
multiple choice questions
Advantages:
The objective test can offer a total feedback
You can check the correctness of each answer. 10

11. Evaluation: Type of test
Simulation test
Definition:
Test where you realize a certain activity, simulation, in which the success or failure can be detected automatically by an application. This is a type of an objective test.
Example:
Flight simulation
Driving Simulation
Advantages:
 Try to play a real business 11

12. Donald Kirkpatrick’s learning evaluation theory
The effect on the business or environment by the trainee
The extent of applied learning back on the job implementation
ROI
The measurement of the increase in knowledge - before and after
Transfer
Knowledge
How the students felt about the training experience
Reaction Layer 12

13. Examples of evaluation tools and methods
Donald Kirkpatrick’s learning evaluation theory
Evaluation type
Examples of evaluation tools and methods
Relevance
and
practicability
Reaction
E.g.., 'happy sheets', feedback forms
also verbal reaction, post-training surveys or questionnaires
Quick and very easy to obtain
Not expensive to gather or to analyze
Knowledge
Typically assessments or tests before and after the training
interview or observation can also be used
Relatively simple to set up; clear-cut for quantifiable skills
Less easy for complex learning
Transfer
Observation and interview over time are required to assess change, relevance of change, and sustainability of change
Measurement of behavior change typically requires cooperation and skill of line-managers
Impact
Measures are already in place via normal management systems and reporting - the challenge is to relate to the trainee
Individually not difficult; unlike whole organization
Process must attribute clear accountabilities 13

14. Thank you for your attention!