Runway Safety Metric (RSM) Presented to: RSM Stakeholders By:

Runway Safety Metric (RSM) Presented to: RSM Stakeholders By: Firdu Bati, AJI-3 Date: March, 2017 Federal Aviation Administration Federal Aviation Administration

Background The primary goal of safety metrics is to gauge the safety performance of the National Airspace System (NAS), at the system and facility levels. Existing runway safety metrics focus solely on runway incursion type counts, not accidents and excursions. RSM Goal: Develop a runway safety metric that includes accident, Runway Excursion (RE), Runway Incursion (RI), and Surface Incident (SI) data. Federal Aviation Administration 2

Approach Collect applicable accident and incident data. – National Transportation Safety Board (NTSB), RI-SI, and RE data sources Use modeling to assign risk weights to all kinds of outcomes. – Weighted outcomes include injuries, damage, incursion types, surface incidents, and excursions – Weights are based on outcomes’ “proximities” to fatality and are ordered based on SME input [Injury Damage Incident] – For accidents, weighting gives some credit for saving lives and minimally-damaged aircraft Aggregate all event weights for each fiscal year to get a FY score. Federal Aviation Administration 3

Weighting Schematic Federal Aviation Administration 4

Combined Risk & Event Count *Risk decreased, incident increased, accident relatively constant over time Federal Aviation Administration 5

Commercial Risk & Event Count *Risk decreased, incident increased, accident relatively constant over time Federal Aviation Administration 6

Non-Commercial Risk & Event Count *Risk decreased, incident increased, accident relatively constant over time Federal Aviation Administration 7

Commercial Risk & Ops (Per Million) *Risk decreased, Ops slightly decreased over time Federal Aviation Administration 8

Non-Commercial Risk & Ops (Per Million) *Risk increased and started decreasing, Ops decreased over time Federal Aviation Administration 9

Conclusion Systematic approach followed to develop Runway Safety metric Inclusive of all relevant datasets Quantitative technique to assign severity weights Baseline/target is in development Expected deployment date: FY 19 Federal Aviation Administration 10

Backup Slides Federal Aviation Administration 11

Technical Approach Prepare National Transportation Safety Board (NTSB), RI-SI Database, and RE Database data. Assume the worst possible accident involves fatal injury. Assign weights to accidents and incidents based on relative “distance” to fatality. Federal Aviation Administration 12

Data Pre-Processing NTSB data selection – Reports without phase codes relevant to the runway environment were removed – Remaining data classified as runway collision, taxiway collision, runway excursion by training a text mining model and validating results Data merging – RE and RI-SI data have already been reviewed by domain expects and categorized – Some overlap between RE and NTSB datasets; duplicates were removed (NTSB records retained) Federal Aviation Administration 13

FY16 Dataset NTSB (FY81 – FY16) Runway Collision Taxiway Collision Runway Excursions 237 302 9,653 # of Events RI-SI (FY98 – FY16) # of Events A B C D E SI 235 319 5,472 7,967 12 8,936 RE (FY12 – FY16) Runway Excursions # of Events 1,276 Federal Aviation Administration 14

Event Count Federal Aviation Administration 15

Three Step Weighting Process 1. Weights are assigned to types of accidents based on proximity to a fatal accident. 2. Appropriate domain expert assumptions are used to reorder the weights: – Injury – Damage – Incident without injury/damage 3. Incidents are assigned weights based on their corresponding accident types. Federal Aviation Administration 16

Information Gain as a Measure of Uncertainty 𝐻 ( 𝐹𝐴) 𝑃 ( 𝑥 ) 𝑙𝑜𝑔2 𝑃, 𝑥 𝑋 𝐻 ( 𝐹𝐴 𝐴𝑐𝑐 ) 𝑃 ( 𝐹𝐴𝑖 ) 𝐻(𝐹𝐴𝑖 𝐴𝑐𝑐 𝑖) 𝑖 𝐼 ( 𝐹𝐴 , 𝐴𝑐𝑐 ) 𝐻 ( 𝐴𝑐𝑐 ) 𝐻(𝐹𝐴 𝐴𝑐𝑐) 𝑝 ( 𝑓𝑎 , 𝑎𝑐𝑐) 𝐼 ( 𝐹𝐴 , 𝐴𝑐𝑐 ) 𝑝 ( 𝑓𝑎 , 𝑎𝑐𝑐 )𝑙𝑜𝑔2 𝑝 ( 𝑓𝑎 ) 𝑝(𝑎𝑐𝑐 ) 𝑓𝑎 𝐹𝐴 𝑎 𝑐𝑐 𝐴𝑐𝑐 Federal Aviation Administration 17

Information Gain as a Measure of Uncertainty Bayesian Network employed – Accounts for correlation between different outcomes – Computes information gain Federal Aviation Administration 18

SME-Based Reordering Logic Model output ranks some damage higher than injury. Domain experts provided qualitative direction resulting in reordering. 1. Injury 2. Damage 3. Incident Federal Aviation Administration 19

SME-Based Reordering Logic – – – – – – Compute information gain (model output) Compute relative significance of highest gain Normalize it so the relative gain sums to 1 Order gain using cumulative significance Compute relative significance to new highest gain Shift significance by maximum gain to compute relative significance to fatality Federal Aviation Administration 20

Entropy-Based Relative Weight Commercial Non-Commercial Federal Aviation Administration 21

Weights – Penalty & Credit Injuries – ( 𝑤 𝐼 𝐼𝑛𝑗 𝐶𝑛𝑡 𝑁𝑜𝑛 𝐼𝑛𝑗𝑢𝑟𝑦 𝐶𝑛𝑡 𝑤𝑖 𝑇𝑜𝑡𝑎𝑙𝑂𝑛𝑏𝑜𝑎𝑟𝑑 Damage – Penalty and credit terms for injured and non-injured, respectively Penalty and credit terms for damaged and non-damaged, respectively ( 𝑤 𝐷 𝐷𝑚𝑔𝐶𝑛𝑡 ) 𝑁𝑜𝑛𝐷𝑚𝑔 𝐶𝑛𝑡 𝑤 𝑑 𝑁𝑢𝑚𝑏𝑒𝑟𝑂𝑓𝐴𝑖𝑟𝑐𝑟𝑎𝑓𝑡 ) Incidents – Ratio of the weight of the corresponding accident type to frequency of incident 𝑤 𝐼𝑛𝑐 𝑤 𝐴𝑐𝑐 𝑛 𝐼𝑛𝑐 , 22 Federal Aviation Administration

Credit – Penalty Correction Term Example Two accidents – A: 5/5 injured – B: 5/200 injured Without credit term, – Both A & B: 5 * .75 3.75 With credit, – A: (5*.75) 3.75 – B; (5*.75) – (.975*.75) 3.02 Federal Aviation Administration 23

Relative Weights – Commercial Note: Y-Axis uses Logarithmic scale Federal Aviation Administration 24

Relative Weights – Non-Commercial Note: Y-Axis uses Logarithmic scale Federal Aviation Administration 25

Application of Weights Two potential applications: – Aggregation: sum of all undesired outcome weights – Worst-outcome: weight of the worst outcome *The choice makes little difference in the relative risk profile primarily due to the limiting effect of the worstoutcome [adding small numbers to a large number] Federal Aviation Administration 26

Combined Risk & Event Count *Risk decreased, incident increased, accident relatively constant over time Federal Aviation Administration 27

Commercial Risk & Event Count *Risk decreased, incident increased, accident relatively constant over time Federal Aviation Administration 28

Non-Commercial Risk & Event Count *Risk decreased, incident increased, accident relatively constant over time Federal Aviation Administration 29

Commercial Risk & Ops (Per Million) *Risk decreased, Ops slightly decreased over time Federal Aviation Administration 30

Non-Commercial Risk & Ops (Per Million) *Risk increased and started decreasing, Ops decreased over time Federal Aviation Administration 31

Assumptions Assumed the following hierarchy of accident/incident severity based on domain expert input: fatality, injury, aircraft damage, incidents. Incident to accident mapping assumption – RIs Runway Collision – SIs Taxiway Collision – RE Incidents RE Accident Assumed the following criteria should be used to split the dataset by Commercial and Non-Commercial – Part 121, 129, 135: Commercial – Others: General Aviation / Non-Commercial Incident data are only available starting in 1997; however, the model used accident data since 1982. – This is more conservative Federal Aviation Administration 32

Conclusion Systematic approach followed to develop Runway Safety metric Inclusive of all relevant datasets Quantitative technique to assign severity weights Baseline/target is in development Federal Aviation Administration 33