1 Data Quality Metric Based on Time-out CNX CDM A&D Meeting 5 June 2001 Dennis Gallus

2 ORD GDP

3 Purpose of Briefing Idea: Encourage improvement in data quality –Perhaps through permission to use slot-hold flag Choose airline-controllable performance measure: Time-out cancellations –Define the metric –Set acceptable standards –Determine fall-out for unacceptable performance Other possible metrics (later): –CNX flights that fly –Pop-up flights (without CDM messages)

4 “First Principles” The number of wasted slots is what we want to reduce (through better data) A TO CNX flight during a GDP is potentially a wasted slot Metric should allow comparison across many airports (but not average over airports) –Normalization required

5 Normalization colors the result 3/25/01ORD Metric: #TO CNX by airline / Total # arrivals by that airline, expressed as a percent Looks like: AALACAAWECOADALFDXNWAOther TWAUALUPSUSA The above metric implies that AWE contributed heavily to the TO CNX problem…

6 Normalization colors result--badly 3/25/01ORD Metric: #TO CNX by airline / Total # arrivals by that airline (%) AALACAAWECOADALFDXNWAOtherTWAUALUPSUSA The above is based on this data: AALACAAWECOADALFDXNWAOtherTWAUALUPSUSA total arrivals T/O CNX In fact, AWE was no worse than three other airlines. UAL was the biggest contributor to TO CNX.

7 Total Arrivals by all airlines at that airport is a better normalization Metric: #TO CNX by airline / Total # arrivals at that airport on that day T/O CNX: AALACAAWECOADALFDXNWAOtherTWAUALUPSUSA total arrivals at ORD on 3/25/01: 1295 Do division and multiply by 1000; round to a single digit to get a nice reference number that translates to “TO CNX per 1000 arrivals” Metric now looks like: AALACAAWECOADALFDXNWAOtherTWAUALUPSUSA This metric more accurately depicts each airline’s contribution to TO CNX problem.

8 Trend--All TO CNX at ORD, Apr Normalized by total # arrivals at ORD each day

9 Trend--All TO CNX at LGA, Apr Normalized by total # arrivals at LGA each day Some airlines have near-perfect days between bad ones...

10 Controlled flights are most important Normalized by # slots in GDP Compare to the previous metric: Normalized by total # Arr at ORD

11 We can eliminate CNX-but-flew... Should CNX-but-flew flights count against airline data quality score?

12 Recommendations Track TO CNX for controlled flights Phase in statistical limit –Perhaps 2 std.dev. at the start –Running calculation, permit x bad days per month? Collectively decide where to put the bar 2 std dev. 1 std dev. Average

13 BACKUP SLIDES

14 Desirable characteristics of data quality metric Simple Airline-controllable Reflect only the carrier being measured Operation-independent Minimally disputable

15 We can eliminate TO CNX flights removed at first ADL update Qualitatively the same as that for all controlled flights

16 Data Quality issues Stale data –Time-out delay and Time-out CNX Spurious data--airlines –Substitution flights given inflated ETE –CNX-that-flew (without proper reinstatement) Spurious data--system –Duplicate ACIDs generated by ETMS –Bad ETEs used in modeling Missing data –No DZ or AZ msgs –OAG flights (FS msg but no FZ) Bad operations on good data –FSM algorithm flaws –Specialist’s failure to enter end time of previous GDP in a revision, causing flights to be sorted by IGTA vice CTA –Other GDP setup panel errors that sub-optimize result Other –Effects of MIT during GDP

17 Metron Aviation Under-delivery studies ERTA submissions MIT/GDP analysis C-Flow SMS/ETA predicting OOOI & ETMS Algorithm changes—flights with ETA <= GDP start Algorithm changes—flights with ARTA <= CTA Compliance Arrivals—Actual vs. AAR GS/GDP Order of Operations Training T0 CNX and data quality metrics

18 Add a bar to indicate tolerable limit average 1 std.dev. 2 std.dev