1. Analysis of usability of scheduling algorithms: a case study in the Netherlands Railways R.J. Jorna, W. van Wezel & J. Bos Faculty of Economics and Business, University of Groningen (Groningen, The Netherlands) Contact: r.j.j.m.jorna@rug.nlr.j.j.m.jorna@rug.nl Third Rail Human Factors Conference; Lille, France, March 3-5, 2009

2. 2 Structure Introduction – Netherlands Railways and their planning – Background & Problem statement: effects of planning support Research: – What does “inadequate use” of algorithms mean? – Results of experiments with 8 planners – Comparison planners and OR-specialist Algorithms and the function-allocation theory Conclusions, statements and further research

3. 3 The Netherlands Railways and Planning Netherlands Railways (NS): 2700 railroad carriages; per day 5000 train services; 384 stations. NS overall: about 350 planners; 4 major kinds of planning: – A: timetables; – B: planning rolling stock and rolling staff; – C: local and central planning (stock and staff); – D: year plan (long term) & day plan (short term) – Sequence: timetable, rolling material, rolling staff and then shifts for individual persons.

4. 4 Background and problem statement In staff planning shift schedules need to be changed frequently (see Figure 1). Here, study of 8 planners of Day Planning (Central) (re)schedule shifts of train drivers/ticket collectors. Planners use DSS CREWS (see Figure 2), with a GUI to create plans by dragging and dropping. Recently, an algorithm to automatically create plans is included. Planners can influence algorithm by setting parameters. Problem: Use of algorithm did not result in expected increase in performance. What is going on?

5. 5 Example of change within shifts Figure 1

6. 6 Figure 2 Screen shot of CREWS

7. 7 Theoretical assumptions: general Task performance with DSS is a function of – Planning problem complexity, – Quality of automation (software support level), – Mental load of planners, – Many other aspects (trust, cost, cooperation, etc.) Use of algorithmic support can be studied in quasi-experiments with planners

8. 8 Cognitive workload Situation awareness Complacency Skill degradation Quality of automation Risk of automation failure Trust Cost of incorrect decisions Cognitive workload Situation awareness Complacency Skill degradation Quality of automation Risk of automation failure Trust Cost of incorrect decisions How can we apply this to planning / scheduling? How can we apply this to planning / scheduling? Factors influencing the adequate use of planning support

9. 9 Theoretical assumptions: model Relation between independent variables and performance Two conditions: manual and algorithm support Two conditions: simple and complex problem

10. 10 Experiments with planners (1) Procedure – Planner reads assignments, solves problem with system; – For Mental Load: planner uses NASA TLX questionnaire; – During experiment, number of activities of planner are counted: a) use of searching functionality in system; b) # backtrackings (withdrawn reasoning steps); # decisions resulting in constraint violations

11. 11 Experiments with planners: results simple problem (2) For simple problem: algorithm better than manual; for complex the reverse Question: WHY? +: algorithm better

12. 12 Experiments with planners: results mental load and complex problem (3) No difference mental load for simple problem (N=8) nor for complex problem (N=2) Experiment with complex problem took more than 8 hours (only two planners were analyzed)

13. 13 Experiments with planners: conclusions (4) Some conclusions: For algorithms, planners are concerned with usability: – does planner understand outcome of algorithm? – can schedule be explained to the staff that will execute it? – can planner change schedule easily, etc.) (all of the above are difficult to formalize) If algorithm is not aligned to planner, no increase of performance In using algorithms, significant individual planner differences. Individual planner differences concerning: mental load, time to solve problem, quality of outcome. Planners spend much time on configuring parameters and interpreting the outcome (from protocols).

14. 14 Experiment comparing planners, algorithm and developer (5) (n=2)Planners without algorithm Planners with algorithmDeveloper Time (minutes)29018560 Extra shifts14127 Empty shifts1286 Number of steps70927127 Passenger tasks564836 Taxi00,57 Conflicts000 Exceed max shift length16113 Planners can not plan as good as OR specialists?

15. 15 What is wrong with planners? Planners do not use their algorithms correctly and should be trained or Algorithms are not developed to be usable by planners (The algorithm was designed by winners of 2008 Informs “Franz Edelman Award for Achievement in OR and the Management Sciences”. Should they, the developers, become planners?) Is planning/scheduling in railways a matter of control of dynamic situations and should support be adjusted to this kind of task?

16. 16 12-02-2008 | 16 CALCULATE DURING THE NIGHT FIX ERRORS

17. 17 Basic assumptions concerning planners in algorithmic support Interaction with algorithms implies knowledge of the model being used, e.g.: – Specify parameters; specify weights on goal functions – Steer the backtracking process of the algorithm – Fix the solution manually How does interaction between parameters lead to constraint violations and goal realization? What goal? – Planners do not have this knowledge !! – Planners will never have this knowledge !! We propose that Interaction should be determined a priori, instead of a posteriori

18. 18 The perspective of the control of dynamic situations “Dynamic situations are situations which are only partly controlled by the human. He/she then has to keep the situation between acceptable limits while managing high risk and/or time pressure” Examples: Air traffic control, nuclear power plant, aircraft operation, flexible manufacturing system In the field of Cognitive Ergonomics, Function Allocation is successfully used to design automation

19. 19 Conclusions / statements The fundamental problems concerning planning support did not change in 30 years Practice is ahead of academia We don’t fully understand ‘old’ planning, yet businesses started ‘new’ planning Organization theory (various organizational forms) related to planning hinders adequate support Algorithms do not pay attention to human problem solving

20. 20 Conclusions / further research Balance requirements of tasks, algorithmic possibilities, and human limitations/creativity Include the user in the development process from the beginning, but: Not necessarily take the limitations of current humans for granted! Training or even partly replacing human planners is an option as well. However, many goals of planning/scheduling are often confusing / difficult to formalize!

21. 21

22. 22 Interactive Scheduling Systems; gap between theory and practice  Combinatorial problems are hard to tackle  Problems in practice change  Interactive systems are rare  Operations managers lack understanding of scheduling systems  Cost/benefit ratio hard to prove  Human planners don’t have a good grasp of the implications of their decisions  Schedulers are buffered from outside pressure

23. 23 The planner and his tasks Macro task division Organizational setting Who does what? Order acceptance, stock planning, scheduling, rescheduling, etc. Micro task division Collecting information Counting Checking errors Evaluating Problem solving