1. 1 Arjan Westerhof 04 June 2016 Implementing an O&D System at KLM March 23, 2000 Arjan Westerhof Agifors yield management study group

2. 2 Arjan Westerhof 04 June 2016 Outline Project overview Short description of all major phases. Main focus on:  Specification  Business tests Current status and concluding remarks

3. 3 Arjan Westerhof 04 June 2016 Project Overview Approach: entirely new core system O&D based data Completely new demand forecasting Completely new fare forecasting Network optimization New hardware New programming methods

4. 4 Arjan Westerhof 04 June 2016 Project Overview 1997199820001999 S p e c i f i c a t i o n D e t a i l e d S p e c i f i c a t i o n a n d u n i t t e s t i n g I n t e g r a t i o n t e s t i n g a n d p e r f o r m a n c e t u n i n g B u s i n e s s t e s t i n g & s y s t e m i m p r o v e m e n t Vendor choiceSpecs agreedUnit test completePerformance +/-OKFirst flight liveSmall network liveBus. test completeLarge network live

5. 5 Arjan Westerhof 04 June 2016 Business testing & system improvement Integration testing and performance tuning Project Overview 1997199820001999 1998 Detailed specification and Unit test Integra- tion test Acc. test Detailed spec. and Unit test

6. 6 Arjan Westerhof 04 June 2016 Specification Complex, state of the art system (PNR based) Why? This is for an airline like KLM the only method that will give accurate short term demand forecasts 1997 1998

7. 7 Arjan Westerhof 04 June 2016 Specification: Real Data Example I Different types of passengers using the same flight and class have different booking curves 1997 1998

8. 8 Arjan Westerhof 04 June 2016 Specification: Real Data Example II Different types of passengers using the same flight and class have different booking curves 1997 1998

9. 9 Arjan Westerhof 04 June 2016 Specification: Simplified Example Different types of passengers using the same flight and class have different booking curves When aggregated data is used, inaccurate forecast of the demand to come will result. Simplified example:

10. 10 Arjan Westerhof 04 June 2016 Specification: Simplified Example 201000 20 10

11. 11 Arjan Westerhof 04 June 2016 Specification: Simplified Example 201000 20 10

12. 12 Arjan Westerhof 04 June 2016 Specification: Simplified Example Using the low level (PNR) data will give the correct forecast But... complex system is much more work than simple system

13. 13 Arjan Westerhof 04 June 2016 Unit Testing Unit testing with self constructed testcases  Limited data in order to be able to determine the expected result with manual or spreadsheet calculation  Constructed in such a way that ‘all’ logical cases are tested Started with input data modules to start buildup of historical O&D data asap  Problems with data quality  Hard to get the tailor made software correct 19971998

14. 14 Arjan Westerhof 04 June 2016 Integration Testing Modules worked quite good together, but… Large amounts of real life data contain strange values of which some were not tested Much more data than expected  Performance problems Redesign for performance (and again unit testing,...) 199719981999

15. 15 Arjan Westerhof 04 June 2016 Business Testing 19971998199920002000 Will the system generate extra revenue?

16. 16 Arjan Westerhof 04 June 2016 Business Testing What? Analyses of: Fare forecasting Demand forecasting Optimization 19971998199920002000

17. 17 Arjan Westerhof 04 June 2016 Business Testing How? Data analyses on the O&D data Comparison with current systems leg data Expert opinion on leg and O&D data 19971998199920002000

18. 18 Arjan Westerhof 04 June 2016 Business Testing: Fare Forecast Percentage of tickets that has certain forecast error. Forecast can be evaluated for: Input data New data %Tkts<=10%tkts<=20% Overallx 1 %y 1 % Top 100 country – countryx 2 %y 2 % Top 20 POSx 3 %y 3 % Top 100 city – cityx 4 %y 4 % Top 20 city – cityx 5 %y 5 % 19971998199920002000

19. 19 Arjan Westerhof 04 June 2016 Business Testing: Fare Forecast Consistency of forecasts (higher subclass should in general have higher fare). %Consistent ranking OverallX 1 % Top 20 country – country X 2 % Top 20 POS X 3 % Top 20 city – city X 4 % 19971998199920002000

20. 20 Arjan Westerhof 04 June 2016 Business Testing: Demand Forecast Do the forecasts match the input data? 19971998199920002000

21. 21 Arjan Westerhof 04 June 2016 Business Testing: Demand Forecast Comparison forecasts with reality (note: reality is constrained, forecast is unconstrained) 19971998199920002000

22. 22 Arjan Westerhof 04 June 2016 Business Testing: Optimization Look how forecasts and bidprices develop in time KL 1024 DEP=LHR ARR=AMS 20-Feb-2000 0 20 40 60 80 100 120 140 160 020406080 Days to departure Seats O&D FORECAST SEATS SOLD BIDPRICE 19971998199920002000 CURRENT SYSTEMS FORECAST

23. 23 Arjan Westerhof 04 June 2016 Business Testing: Optimization Comparison of overbooking levels 19971998199920002000

24. 24 Arjan Westerhof 04 June 2016 Current Status Small network live Not yet completely happy with the results still working on system improvement Expect to implement major improvements in April (currently in unit testing) 19971998199920002000

25. 25 Arjan Westerhof 04 June 2016 Lessons Learned Doing everything at the same time has some advantages, but a more gradual approach might be better Not recommended to implement new system on new hardware Everything takes much longer than expected The time needed to get from a running system to a system that generates business value is very long Tough project with various parties Complex system makes all the above things harder but is the only way generate the promised revenue. 1997199819992000

26. 26 Arjan Westerhof 04 June 2016 Questions ?