1. Bas Koene (course coordinator) Dept 2 - Organization and HRM
Peer reviewing of case study assignments by 2nd year Ba students Course: Organization Theory and Dynamics
Bas Koene
(course coordinator)
Dept 2 - Organization and HRM

2. Introduction Organization theory
Theory not difficult
But application:
Apply carefully and precisely
Deal with real-life ambiguity in assessment and analysis
Relevance clear from practical application
Course OTD:
900 students, mandatory introductory course
Lectures: overview organization theory
Learning Communities: support case research and writing
2 lecturers, 6 LC coaches, 2 stud assistants

3. Learning communities 1200 students 24 learning communities
50 students per community
10 teams of 5 students
7 LC meetings: interaction between teams
Poster presentations, short plenary presentations, role plays
Feedback on own work, ideas from what other teams do, practice skills (presentation, feedback, dialogue)
Peer review process

4. Peer review process
Final case study assignment  build up in three parts
General description of organization
Investigation culture, photo ethnography
Interviewing about change objectives and methods
Peer reviews:
Active engagement with work other teams (see different solutions)
Creativity and benchmarking own progress
Engage with theory and reports in different way

5. Peer review system: Expertiza
Automated (dealing with 900 plus students)
Ability to give qualitative feedback, comments for improvement (not just scoring)
Meta-feedback possible
Get feedback on your review
Peer control, signaling function for teacher
Aggregate scores (feedback and meta-feedback available for teacher)

6. Expertiza Pilot: who involved?
Surf-net: comparison peer reviews systems
RISBO / American developers
Comparison systems (surf-net), selection of system
Setting it up for use / adaptation for us
User manuals students / teaching staff & stud assistants
Support in first use / specialist ‘helpdesk’
BIT: technical implementation
Department
Academic staff: review questions
Student assistant(s!): maintain system, first line helpdesk, etc
Secretariat: back up…

7. Implementation program
Jan 08 – surf-net seminar peer reviewing, some non-teaching examples
Feb 08 – talks with RISBO about possibilities
May 08 – tentative choice for Expertiza, RISBO contact with US
June 08 – test version at RSM, adaptations made for us
July 08 – testing of system
Aug 08 – preparation of system for use
Sep 08 – first use of system

8. Ambitions/expectations – experience (1)
Structured peer review in depth feedback
Steer by structuring process, not outcomes
Feedback helpful for improving assignments
Three reviews will safeguard some good feedback
Possibilities for grading?
Initial experiences:
Need to learn to work with giving feedback
Know how to work with theory in analysis
Value of being critical in a constructive way
Structured feedback list works
But need for staff feedback (as example / to set standard) early on or need less ambitious feedback lists?
Students asked for peer reviews in second round
First round sep: 1 in 3 reviews really useful, but: good reviews take time!
Quality numerical extremely (!) varied.
Some expertiza issues:
Feedback screens could be clearer
Not used overview facilities to check to progress and quality of peer reviews

9. Ambitions/expectations – experience (2)
Use independently of Learning communities
First presentations in LCs
Afterwards peer reviewing with additional insight, better reviews?
Anonymity needed?
Students too nice to each other?
Students not serious, just going through the motions
Deadlines necessary?
Use Expertiza to monitor peer review process
Teaching staff final evaluations also through system
Initial perceptions:
Need for face-to-face contact after review
Peer control, leverage reviews in dialogue (Not anonymous!!)
Work with paper and review comments immediately after receiving them
Good students extremely clear about bad (light) feedback
Yes, how set right ambition level for papers without direct teacher feedback?
Not reading assignments / tasks disrupts online interaction! (interdependence teams)
Yes! Clear deadlines are necessary
Pilot: fine-tuning system more work,
monitoring face-to-face in my own LCs
Expertiza interface not efficient if teacher wants to monitor quality of feedback
Make questions that lead to comprehensive feedback ‘stories’
Organize feedback interface differently (by question not by respondent
Need to do all grading off-line first, need overview scores in grading process

10. What do you need to make it work?
Our constraints:
Expertiza Pilot: first year
Large course: (1200+ students)
Need dedicated help-desk for questions and for dealing with technical issues and deadlines (Have enough manpower available. At least two student assistants!)
Extremely busy around deadlines (do not set them at 17.00!)
Need way to deal with exceptions (what if teams are late uploading their paper / reviewing)?
FAQ system would be useful. Course addresses + BB facility worked, discussion board becomes to messy
Guard simplicity of tasks and requirements. Way to remind them…
Pilot:
The proof of the pudding…
Hotline with professional support very useful! (in our case BIT and RISBO)
How handle unexpected: Students appreciate information and feedback in case of unexpected events