Bas Koene (course coordinator) Dept 2 - Organization and HRM Peer reviewing of case study assignments by 2nd year Ba students Course: Organization Theory and Dynamics Bas Koene (course coordinator) Dept 2 - Organization and HRM

Introduction Organization theory Theory not difficult But application: Apply carefully and precisely Deal with real-life ambiguity in assessment and analysis Relevance clear from practical application Course OTD: 900 students, mandatory introductory course Lectures: overview organization theory Learning Communities: support case research and writing 2 lecturers, 6 LC coaches, 2 stud assistants

Learning communities 1200 students 24 learning communities 50 students per community 10 teams of 5 students 7 LC meetings: interaction between teams Poster presentations, short plenary presentations, role plays Feedback on own work, ideas from what other teams do, practice skills (presentation, feedback, dialogue) Peer review process

Peer review process Final case study assignment  build up in three parts General description of organization Investigation culture, photo ethnography Interviewing about change objectives and methods Peer reviews: Active engagement with work other teams (see different solutions) Creativity and benchmarking own progress Engage with theory and reports in different way

Peer review system: Expertiza Automated (dealing with 900 plus students) Ability to give qualitative feedback, comments for improvement (not just scoring) Meta-feedback possible Get feedback on your review Peer control, signaling function for teacher Aggregate scores (feedback and meta-feedback available for teacher)

Expertiza Pilot: who involved? Surf-net: comparison peer reviews systems RISBO / American developers Comparison systems (surf-net), selection of system Setting it up for use / adaptation for us User manuals students / teaching staff & stud assistants Support in first use / specialist ‘helpdesk’ BIT: technical implementation Department Academic staff: review questions Student assistant(s!): maintain system, first line helpdesk, etc Secretariat: back up…

Implementation program Jan 08 – surf-net seminar peer reviewing, some non-teaching examples Feb 08 – talks with RISBO about possibilities May 08 – tentative choice for Expertiza, RISBO contact with US June 08 – test version at RSM, adaptations made for us July 08 – testing of system Aug 08 – preparation of system for use Sep 08 – first use of system

Ambitions/expectations – experience (1) Structured peer review in depth feedback Steer by structuring process, not outcomes Feedback helpful for improving assignments Three reviews will safeguard some good feedback Possibilities for grading? Initial experiences: Need to learn to work with giving feedback Know how to work with theory in analysis Value of being critical in a constructive way Structured feedback list works But need for staff feedback (as example / to set standard) early on or need less ambitious feedback lists? Students asked for peer reviews in second round First round sep: 1 in 3 reviews really useful, but: good reviews take time! Quality numerical extremely (!) varied. Some expertiza issues: Feedback screens could be clearer Not used overview facilities to check to progress and quality of peer reviews

Ambitions/expectations – experience (2) Use independently of Learning communities First presentations in LCs Afterwards peer reviewing with additional insight, better reviews? Anonymity needed? Students too nice to each other? Students not serious, just going through the motions Deadlines necessary? Use Expertiza to monitor peer review process Teaching staff final evaluations also through system Initial perceptions: Need for face-to-face contact after review Peer control, leverage reviews in dialogue (Not anonymous!!) Work with paper and review comments immediately after receiving them Good students extremely clear about bad (light) feedback Yes, how set right ambition level for papers without direct teacher feedback? Not reading assignments / tasks disrupts online interaction! (interdependence teams) Yes! Clear deadlines are necessary Pilot: fine-tuning system more work, monitoring face-to-face in my own LCs Expertiza interface not efficient if teacher wants to monitor quality of feedback Make questions that lead to comprehensive feedback ‘stories’ Organize feedback interface differently (by question not by respondent Need to do all grading off-line first, need overview scores in grading process

What do you need to make it work? Our constraints: Expertiza Pilot: first year Large course: (1200+ students) Need dedicated help-desk for questions and for dealing with technical issues and deadlines (Have enough manpower available. At least two student assistants!) Extremely busy around deadlines (do not set them at 17.00!) Need way to deal with exceptions (what if teams are late uploading their paper / reviewing)? FAQ system would be useful. Course email addresses + BB email facility worked, discussion board becomes to messy Guard simplicity of tasks and requirements. Way to remind them… Pilot: The proof of the pudding… Hotline with professional support very useful! (in our case BIT and RISBO) How handle unexpected: Students appreciate information and feedback in case of unexpected events