2. 1 Arroyo Center RAND UXO Risk Assessment Methods: Critical Review Jacqueline MacDonald, Debra Knopman, J. R. Lockwood, Gary Cecchine, Henry Willis RAND

3. 2 Arroyo Center RAND Briefing Outline  Need for UXO risk assessment methods Prioritization Site-specific assessment  RAND review of existing methods: tasks, approach  Design features and limitations of existing methods  RAND recommendations for improving UXO risk assessment Prioritization Site-specific assessment

4. 3 Arroyo Center RAND Need for UXO Risk Assessment  Prioritization: mandated by Congress  Site-specific assessment Cost differences among alternative response options are significant Total current cost estimate of $14 billion assumes one scan of site and excavation of anomalies only Alternative approaches proposed by regulatory agencies could cost much more DOD needs to understand risk reduction differences among alternative approaches

5. 4 Arroyo Center RAND Alternatives for UXO Response  Surface clearance only  Scan with metal detector, and excavate each anomaly to a specific depth (e.g. 2 ft, 3 ft)  Scan, excavate each anomaly, scan bottom of hole, and excavate again if anomaly is detected  Scan and excavate anomalies, and then repeat the process two or more times  Excavate the entire site in one-foot lifts to depths of 2 ft, 4 ft, or more; sift the excavated soil to remove UXO

6. 5 Arroyo Center RAND Alternatives Proposed at Ft. X Scan land (surface & 1ft)Excavate entire site to 2 ft Scan land (2ft level)Excavate entire site to 3 ft Scan land (3ft level)Excavate entire site to 4 ft Excavate anomalies found Scan land (4ft level)Excavate anomalies found Scan land with best available technology (at surface, one scan effort) Provide construction support to property developers Excavate all anomalies found to maximum depth (4 feet) Army’s Preferred Approach State EPA’s Preferred Approach

7. 6 Arroyo Center RAND Cost Differences Among Ft. X Options Are Hundreds of Millions of Dollars

8. 7 Arroyo Center RAND Briefing Outline  Need for UXO risk assessment methods Prioritization Site-specific assessment  RAND review of existing methods  Design features and limitations of existing methods  RAND recommendations for improving UXO risk assessment Prioritization Site-specific assessment

9. 8 Arroyo Center RAND RAND Tasks Client: Army Chief of Staff for Installation Management Tasks: 1.Conduct a preliminary analysis of ongoing efforts in UXO risk assessment, including: Ordnance and Explosives Cost-Effectiveness Risk Tool (OECert) Interim Range Rule Risk Methodology (IR3M) Ordnance and Explosives Risk Impact Analysis (very similar to IR3M) Risk Assessment Code (RAC)

10. 9 Arroyo Center RAND Tasks, continued 2.Study methods used by the Department of Energy, National Aeronautics and Space Administration, and others to evaluate and measure risk of low-probability and high- consequence events. 3.Recommend how the Army could develop a risk assessment/risk management protocol for UXO sites.

11. 10 Arroyo Center RAND Study Approach 1.Develop criteria for a technically sound risk assessment, based on risk assessment literature survey and consultations with experts 2.Read all documentation for available methods; test software if available 3.Interview method developers 4.Evaluate extent to which each method satisfies the evaluation criteria

12. 11 Arroyo Center RAND Evaluation Criteria Are in Three Categories  Risk calculation features, e.g. Is the logic sound? Are assumptions reasonable? Is the output reproducible?  Implementation features, e.g. Is the method practical for intended use? Are instructions clear? Is it adaptable to different sites?  Communication features, e.g. Can stakeholders provide input about assumptions, scenarios? Is the method transparent?

13. 12 Arroyo Center RAND Briefing Outline  Need for UXO risk assessment methods Prioritization Site-specific assessment  RAND review of existing methods  Design features and limitations of existing methods  RAND recommendations for improving UXO risk assessment Prioritization Site-specific assessment

14. 13 Arroyo Center RAND Interim Range Rule Risk Methodolgy (IR3M) Purpose: To provide “a comprehensive process for managing, assessing, and communicating risks on former ranges” Four tools:  Explosives Safety Risk Tool  Other Constituent Risk Tool for Human Health  Ecological Risk Tool  Comparative Analysis Tool

15. 14 Arroyo Center RAND Explosives Safety Risk Tool Provides a risk value on a dimensionless scale of A to E The A to E value is determined from decision rules that consider three input factors: 1. Accessibility (A) 2. Overall hazard (O) 3. Exposure (E)

16. 15 Arroyo Center RAND Explosive Safety Risk Tool Output A Risk ValueAccessibilityOverall HazardExposure A< 2< 3< 2 B < 5< 2 < 3 C< 4< 3< 4 < 3< 5< 3 = 5< 3< 2 < 3= 5 D< 4< 5< 4 < 5< 3< 5 E OE

17. 16 Arroyo Center RAND RAND Evaluation of Tool  Considers main factors that determine explosion risks from UXO: hazard severity and likelihood of exposure  However, these factors are poorly quantified  The risk calculation features do not satisfy the RAND evaluation criteria:  Interactions among risk contributors are poorly modeled  Output can be illogical  Assumptions often are not explained  Uncertainties are not described  Output is not reproducible

18. 17 Arroyo Center RAND Hypothetical Example Illustrates IR3M Weaknesses Conditions for hypothetical site: Climatic conditions are such that buried UXO is unlikely to surface Significant digging at the site is unexpected All other conditions are such that O = 5, E = 2 Two UXO response options: Option 1: Excavate UXO to 0.99 ft of depth Option 2: Excavate UXO to 1.01 ft of depth

19. 18 Arroyo Center RAND Calculate A for Options 1 and 2 Depth below land surface 1) All UXO > 10 feet 2) All UXO > 4 feet 3) All UXO > 2 feet 4) All UXO > 1 foot 5) Any UXO < 1 foot Migration /Erosion Potential 1) Very stable 2) Minor migration 3) Moderate migration 4) Significant migration 5) Highly dynamic Intrusion Level of Activity 1)Non-intrusive: surface only 2) Minor: to 1 ft 3) Moderate: to 2 ft 4) Significant: to 4 ft 5) Highly intrusive Option 1: A = 5 Option 2: A = 3 ADepthMigrationIntrusion 1= 1< 2 2= 1< 5 = 2< 3 3= 2< 5 = 3< 4 = 4< 2 4< 4< 5 5= 5< 5

20. 19 Arroyo Center RAND IR3M Output for Options 1 and 2 Is Not Logical A = 5 Risk ValueAccessibilityOverall HazardExposure A< 2< 3< 2 B < 5< 2 < 3 C< 4< 3< 4 < 3< 5< 3 = 5< 3< 2 < 3= 5 D< 4< 5< 4 < 5< 3< 5 E O = 5E = 2A = 3

21. 20 Arroyo Center RAND Participants in IR3M Test Exercise Reached Similar Conclusions IR3M was tested by two independent teams in a hypothetical site evaluation exercise The teams found problems similar to those encountered by RAND: “Intuitively, the scores did not make sense.... If action removes all known UXO, the tool does not reflect a change in protectiveness and therefore there is a flaw in the tool.” “With a 5 on Overall Hazard and Accessibility, reducing exposure does not alter the overall risk. If there is one hand grenade at land surface, the overall risk is E. Thus, there is no way to reduce risk by reducing exposure.”

22. 21 Arroyo Center RAND Army Science Board (1998) Also Reached Similar Conclusions “The existing R3M model is not sufficiently grounded or developed to accompany promulgation of the Range Rule in FY99.” “Do not highlight the existing risk formulation in working toward promulgation of the Range Rule.”

23. 22 Arroyo Center RAND Ordnance and Explosives Cost- Effectiveness Risk Tool (OECert) Purpose: “The... OECert... provides the capability to estimate risks and cost associated with given levels of remediation and to prioritize among any set of sites by life cycle cost and the amount of reduction in public risk expected to be realized from the remediation process.”

24. 23 Arroyo Center RAND OECert Risk Calculation Three methods: 1. Dispersed sector method (ranges, impact areas) 2. Localized sector method (burial pits, trenches) 3. Water sector method (submerged UXO) For all methods: Risk = (Number of Expected UXO Exposures) x (UXO Hazard Factor)

25. 24 Arroyo Center RAND Dispersed Sector Method: UXO Hazard Factor UXO TypeSensitivity Factor Consequence Factor ProductAdjusted Hazard Factor UXO1268010,08029 UXO light motion sensitive 3278026,16076 UXO white phosphorus 126364,53613 Controlled chemical, biological, and radiological 12627334,398100

26. 25 Arroyo Center RAND Dispersed Sector Method: Exposure Calculation  Site is divided into homogeneous “sectors”  UXO density is assumed homogeneous in each sector  Density is calculated using statistical sampling OR based on densities of similar sites in a database of Formerly Utilized Defense Sites  Within each sector, number of exposures to UXO is calculated for 19 activities

27. 26 Arroyo Center RAND Dispersed Sector Method Considers 19 Activities Activities with Risks of Surface Exposures Only Activities with Risk of Surface and Subsurface Exposure HikingFishingChild playingCrop farming Horseback riding SwimmingOff-road driving Archaeology HuntingSurveyingPicnicking Mountain biking RanchingCamping Motor bikingJoggingMetal detection Short cutsConstruction

28. 27 Arroyo Center RAND Example: Camping Total number of exposures is calculated for  hiking to campsite  setting up camp Number of expected campers is determined from  demographic data for surrounding area  generic (nation-wide) estimates of percentage of people in each age category who camp

29. 28 Arroyo Center RAND Exposures During Hiking to Campsite  Distance of path to campsite is estimated as radius of a circle with area equivalent to area of sector  Area traversed by single individual: traversed area = (sector area/π) 1/2 x (path width)  Number of UXO exposures for single individual: exposures = (traversed area) x (surface density)

30. 29 Arroyo Center RAND Exposures At Campsite  Assumed area of campsite = 2,500 ft 2  Assumed subsurface intrusive area = 0.75 ft 2  Exposures within campsite: exposures = 2,500 ft 2 x (surface density) + 0.75 ft 2 x (subsurface density)

31. 30 Arroyo Center RAND Total Exposures for UXO Site

32. 31 Arroyo Center RAND RAND Evaluation of OECert  Overall structure of modeling UXO explosion risk has merit (exposure probability times exposure consequences)  However, there are major limitations:  Contains many unjustified assumptions (e.g., campsite size of 2,500 ft 2, camper intrusion area of 0.75 ft 2 )  Army staff report that output is usually 100,000 exposures  Software crashed when we tried to use it  Output is not meaningful outside OECert context  Process is not transparent  Density estimation procedures need to be improved

33. 32 Arroyo Center RAND Risk Assessment Code (RAC) Purpose: Prioritize sites in the Formerly Utilized Defense Sites (FUDS) program based on UXO explosion risk Two factors considered: 1. Hazard severity 2. Hazard probability

34. 33 Arroyo Center RAND Hazard Severity Score Based on five broad categories of ordnance:  Conventional ordnance and ammunition  Pyrotechnics  Bulk high explosives (uncontainerized)  Bulk propellants  Chemical warfare materiel and radiological weapons A score is developed for each category based on types of ordnance present Total score is based on sum of scores for the 5 categories (maximum possible is 61)

35. 34 Arroyo Center RAND Example: Conventional Ordnance Score Ordnance TypeScore Medium/large caliber (20 mm and larger)10 Bombs, explosive10 Grenades, hand or rifle, explosive10 Landmine, explosive10 Rockets, guided missile, explosive10 Detonators, blasting caps, fuzes, boosters, bursters 6 Bombs, practice (w/spotting charges)6 Grenades, practice (w/spotting charges)4 Landmine, practice (w/spotting charges)4 Small arms, complete round (.22 cal-.50 cal)1 Largest single value is chosen

36. 35 Arroyo Center RAND Hazard Probability Score Based on five subscores related to probability of human exposure to UXO: 1. UXO location (surface, subsurface, etc.) 2. Distance to nearest inhabited location 3. Number of buildings within 2-mile radius 4. Types of buildings (schools, residential, industrial, etc.) 5. Site accessibility (barriers, guards, etc.) 6. Site dynamics (e.g., erosion, future development) Total score is sum of these six subscores (maximum possible is 30)

37. 36 Arroyo Center RAND RAC Score Is on 1-5 Scale Hazard Severity Hazard Probability Frequent (>26) Probable (21-26) Occasional (15-20) Remote (8-14) Improbable (<8) Catastrophic (>20) 11234 Critical (10- 20) 12335 Marginal (5- 9) 23445 Negligible (1-4) 34455

38. 37 Arroyo Center RAND RAND Evaluation of RAC  Model of estimating risk based on exposure probability and explosion severity is sound  Clear, easy-to-use instructions  Appropriate for intended use (priority setting)  Not detailed enough for site-specific assessment  Limitations  UXO density is not considered in the score  Does not address uncertainty  Does not address issues of stakeholder involvement and communication

39. 38 Arroyo Center RAND Summary: Existing Methods Do Not Satisfy Criteria OE- CERT Communication features Implementation features Risk calculation features OERIAIR3MRAC = does not satisfy evaluation criteria = satisfies some criteria = satisfies all criteria

40. 39 Arroyo Center RAND Briefing Outline  Need for UXO risk assessment methods Prioritization Site-specific assessment  RAND review of existing methods  Design features and limitations of existing methods  RAND recommendations for improving UXO risk assessment Prioritization Site-specific assessment

41. 40 Arroyo Center RAND We Recommend a Two-Stage Process for Prioritization Two stages:  Stage 1: sort sites into explosion risk “bins”  Stage 2: within bins, sort by munitions constituent risks Key design feature:  Preserves separate information about explosion, other constituent risks  Separation is necessary; attempts to combine these risks involve value judgments that should be explicit and transparent in decision-making

42. 41 Arroyo Center RAND Munitions constituent risk No explosion risk sites Low explosion risk sites Moderate explosion risk sites High explosion risk sites Determine explosion risk Determine munitions constituent risk for each site in category 1 2 Rank Explosion risk None Low Moderate High Munitions constituent risk Policymakers decide how to allocate resources among bins Modified RAC? Hazard Ranking System or Relative Risk Site Evaluation Primer

43. 42 Arroyo Center RAND Briefing Outline  Need for UXO risk assessment methods Prioritization Site-specific assessment  RAND review of existing methods  Design features and limitations of existing methods  RAND recommendations for improving UXO risk assessment Prioritization Site-specific assessment

44. 43 Arroyo Center RAND We Searched Other Agencies for Models AgencyModel Applicability to UXO EPARisk Assessment Guidance for Superfund Munitions constituent risks NRCProbabilistic risk assessment (PRA) Explosion risks NASAPRAExplosion risks FAAPRAExplosion risks Army (Chemical Stockpile Disposal) PRAExplosion risk

45. 44 Arroyo Center RAND Example: Army Uses PRA to Assess Risk of Chemical Weapons Release Fault tree used in PRA for Tooele Chemical Agent Disposal Facility

46. 45 Arroyo Center RAND Potential Benefits of Using PRA for Explosion Risk Assessment  Widely and successfully used by other agencies and private industry  Could allow for meaningful quantitative evaluation of risks and benefits of response alternatives  Requires systematic analysis of elements of risk, which provides formal structure for decision- making and negotiations

47. 46 Arroyo Center RAND Army Science Board Made Similar Recommendation  Army Science Board reviewed IR3M in October 1998  Army Science Board recommended that the DOD and Army:  “consider the probabilistic and step (I.e., ON or OFF) nature of explosive events and consequences”  “consider using an approach analogous to the successful formulation of design codes based on the probability of structural failure”

48. 47 Arroyo Center RAND Summary of Recommended Approach for Site-Specific Risk Assessment 1.Use EPA Risk Assessment Guidance for Superfund methods to assess risks of munitions constituents 2.Develop probabilistic risk assessment method specific to UXO to assess explosion risks Development should be overseen by a technical advisory committee The method should be independently peer reviewed Template “trees” should be developed Trees then could be modified at individual sites, with substantial stakeholder input

49. 48 Arroyo Center RAND Summary  Existing methods for UXO risk assessment do not satisfy criteria for technical credibility  New prioritization method should include a two- tier screen: Tier 1: sort by explosion risks (using new method—possibly a modified RAC) Tier 2: sort by constituent risks (using existing method from EPA or Defense Environmental Restoration Program)  New site-specific assessment method should Use EPA’s RAGS for constituent risks Use new PRA method for explosion risks

50. 49 Arroyo Center RAND “Fear of harm ought to be proportional not merely to the gravity of the harm, but also to the probability of the event.” --Antoine Arnauld, 1660