1 Arjan Westerhof 04 June 2016 Implementing an O&D System at KLM March 23, 2000 Arjan Westerhof Agifors yield management study group

2 Arjan Westerhof 04 June 2016 Outline Project overview Short description of all major phases. Main focus on:  Specification  Business tests Current status and concluding remarks

3 Arjan Westerhof 04 June 2016 Project Overview Approach: entirely new core system O&D based data Completely new demand forecasting Completely new fare forecasting Network optimization New hardware New programming methods

4 Arjan Westerhof 04 June 2016 Project Overview S p e c i f i c a t i o n D e t a i l e d S p e c i f i c a t i o n a n d u n i t t e s t i n g I n t e g r a t i o n t e s t i n g a n d p e r f o r m a n c e t u n i n g B u s i n e s s t e s t i n g & s y s t e m i m p r o v e m e n t Vendor choiceSpecs agreedUnit test completePerformance +/-OKFirst flight liveSmall network liveBus. test completeLarge network live

5 Arjan Westerhof 04 June 2016 Business testing & system improvement Integration testing and performance tuning Project Overview Detailed specification and Unit test Integra- tion test Acc. test Detailed spec. and Unit test

6 Arjan Westerhof 04 June 2016 Specification Complex, state of the art system (PNR based) Why? This is for an airline like KLM the only method that will give accurate short term demand forecasts

7 Arjan Westerhof 04 June 2016 Specification: Real Data Example I Different types of passengers using the same flight and class have different booking curves

8 Arjan Westerhof 04 June 2016 Specification: Real Data Example II Different types of passengers using the same flight and class have different booking curves

9 Arjan Westerhof 04 June 2016 Specification: Simplified Example Different types of passengers using the same flight and class have different booking curves When aggregated data is used, inaccurate forecast of the demand to come will result. Simplified example:

10 Arjan Westerhof 04 June 2016 Specification: Simplified Example

11 Arjan Westerhof 04 June 2016 Specification: Simplified Example

12 Arjan Westerhof 04 June 2016 Specification: Simplified Example Using the low level (PNR) data will give the correct forecast But... complex system is much more work than simple system

13 Arjan Westerhof 04 June 2016 Unit Testing Unit testing with self constructed testcases  Limited data in order to be able to determine the expected result with manual or spreadsheet calculation  Constructed in such a way that ‘all’ logical cases are tested Started with input data modules to start buildup of historical O&D data asap  Problems with data quality  Hard to get the tailor made software correct

14 Arjan Westerhof 04 June 2016 Integration Testing Modules worked quite good together, but… Large amounts of real life data contain strange values of which some were not tested Much more data than expected  Performance problems Redesign for performance (and again unit testing,...)

15 Arjan Westerhof 04 June 2016 Business Testing Will the system generate extra revenue?

16 Arjan Westerhof 04 June 2016 Business Testing What? Analyses of: Fare forecasting Demand forecasting Optimization

17 Arjan Westerhof 04 June 2016 Business Testing How? Data analyses on the O&D data Comparison with current systems leg data Expert opinion on leg and O&D data

18 Arjan Westerhof 04 June 2016 Business Testing: Fare Forecast Percentage of tickets that has certain forecast error. Forecast can be evaluated for: Input data New data %Tkts<=10%tkts<=20% Overallx 1 %y 1 % Top 100 country – countryx 2 %y 2 % Top 20 POSx 3 %y 3 % Top 100 city – cityx 4 %y 4 % Top 20 city – cityx 5 %y 5 %

19 Arjan Westerhof 04 June 2016 Business Testing: Fare Forecast Consistency of forecasts (higher subclass should in general have higher fare). %Consistent ranking OverallX 1 % Top 20 country – country X 2 % Top 20 POS X 3 % Top 20 city – city X 4 %

20 Arjan Westerhof 04 June 2016 Business Testing: Demand Forecast Do the forecasts match the input data?

21 Arjan Westerhof 04 June 2016 Business Testing: Demand Forecast Comparison forecasts with reality (note: reality is constrained, forecast is unconstrained)

22 Arjan Westerhof 04 June 2016 Business Testing: Optimization Look how forecasts and bidprices develop in time KL 1024 DEP=LHR ARR=AMS 20-Feb Days to departure Seats O&D FORECAST SEATS SOLD BIDPRICE CURRENT SYSTEMS FORECAST

23 Arjan Westerhof 04 June 2016 Business Testing: Optimization Comparison of overbooking levels

24 Arjan Westerhof 04 June 2016 Current Status Small network live Not yet completely happy with the results still working on system improvement Expect to implement major improvements in April (currently in unit testing)

25 Arjan Westerhof 04 June 2016 Lessons Learned Doing everything at the same time has some advantages, but a more gradual approach might be better Not recommended to implement new system on new hardware Everything takes much longer than expected The time needed to get from a running system to a system that generates business value is very long Tough project with various parties Complex system makes all the above things harder but is the only way generate the promised revenue

26 Arjan Westerhof 04 June 2016 Questions ?