1. An Introduction to CUBE and the Navigation Surface R.M. Hare July 2004

2. An intro to What?  CUBE = Combined Uncertainty and Bathymetry Estimator. Developed by Dr. Brian Calder – Electrical Engineer with a background in digital signal processing  Navigation Surface – a database of those bathymetry and uncertainty estimates that can be used to generate chart and ENC soundings and contours (safety-biased) in an automated way. Developed by Lt. Sheppard Smith – NOAA Corps officer Masters thesis project  University of New Hampshire, Center for Coastal and Ocean Mapping and Joint Hydrographic Center (NOAA, Larry Mayer, Andy Armstrong)

3. Why the need?  Most promising developments in hydrography in at least a decade  NOAA Workshop at IOS July 28-29  CUBE will be implemented in HIPS 6.0 It’s already in Fledermaus 6.1 It’s already in Fledermaus 6.1  Navigation Surface will be part of CARIS toolbox for creating field sheet/chart soundings and contours (safety biased)

4. So what is CUBE?  a fresh attempt to “tell the truth” about data acquired during a multibeam surveys  Dave Well’s uses 7 axioms to describe what CUBE is all about  There are 8 steps required (also according to Dave Wells)

5. The 7 axioms of CUBE 1. Most data, most of the time, are mostly okay 2. A sounding is not a depth 3. The appropriate question to ask is “What is the depth (and its uncertainty) here?” NOT “How good is this sounding?” 4. Data cleaning is a Triage Operation 5. Every sounding solution must include estimates of its uncertainty 6. MBES data density and spatial resolution may not be sufficient to capture all bathymetric features of interest 7. Models do not represent truth

6. Axiom 1  Most data, most of the time, are mostly okay has two practical consequences: has two practical consequences: 1.Blunders (outliers, inappropriate soundings) exist, but not commonly 2.Systematic uncertainties have been removed (to the extent possible) in advance of applying CUBE

7. Axiom 2  A sounding is not a depth highlights the fallacy of the “golden sounding”, based on the belief that soundings are “real” highlights the fallacy of the “golden sounding”, based on the belief that soundings are “real” All soundings are results of measurements (e.g. of time, of sound speed, of tide, …) All soundings are results of measurements (e.g. of time, of sound speed, of tide, …)

8. Axiom 3 (my paraphrase)  Mariners want to know: “What is the depth (and its uncertainty) where I am?” NOT “How good is that (selected) sounding over there?” intent of data cleaning is the estimation of derived parameters (depths at nodes), rather than assessment / flagging of measured soundings intent of data cleaning is the estimation of derived parameters (depths at nodes), rather than assessment / flagging of measured soundings what we call “soundings” these days are themselves estimated derived parameters, based on a multitude of measurements (two-way travel time, beam direction, roll, pitch, heave, dynamic draft, etc.), and processes (ray tracing using an imperfectly known sound speed profile, etc.) what we call “soundings” these days are themselves estimated derived parameters, based on a multitude of measurements (two-way travel time, beam direction, roll, pitch, heave, dynamic draft, etc.), and processes (ray tracing using an imperfectly known sound speed profile, etc.)

9. Axiom 4  Data cleaning is a Triage Operation, so… provide power tools to reduce workload provide power tools to reduce workload replace subjectivity with objectivity replace subjectivity with objectivity algorithm should produce repeatable results algorithm should produce repeatable results Same cannot be said about human hydrographic data cleanersSame cannot be said about human hydrographic data cleaners Algorithmic Hydrography tools always divide data into: a) clearly good; Algorithmic Hydrography tools always divide data into: a) clearly good; b) clearly bad; and c) questionable - requiring further attention

10. Axiom 5  Every sounding solution must include estimates of its uncertainty essential to any form of uncertainty propagation (including CUBE) essential to any form of uncertainty propagation (including CUBE) key feature of IHO S-44, 4 th Edition, and of the ISO 191xx family of standards key feature of IHO S-44, 4 th Edition, and of the ISO 191xx family of standards CUBE demonstrates how this information can be exploited to our advantage CUBE demonstrates how this information can be exploited to our advantage

11. Axiom 6  MBES data density and spatial resolution may not be sufficient to capture all bathymetric features of interest you can’t make a silk purse out of a sow’s ear you can’t make a silk purse out of a sow’s ear there will always be features (sand ripples, scours, pebbles, boulders) that are too small for the MBES to detect there will always be features (sand ripples, scours, pebbles, boulders) that are too small for the MBES to detect

12. Axiom 7  Models do not represent truth simplified description of the complexity of the real world simplified description of the complexity of the real world estimation models are based on mathematical processes estimation models are based on mathematical processes Sometimes human experience and gut feelings are more valid than model results Sometimes human experience and gut feelings are more valid than model results A prudent navigator never depends upon a single source of information A prudent navigator never depends upon a single source of information

13. The 8 steps of CUBE 1. Establish a network of “nodes” or “depth estimation points” over the survey area 2. At each node, maintain sequentially-determined estimates of depth, and depth uncertainty (variance) 3. Compare incoming sounding measurements against these existing nodal estimates 4. Establish a new “hypothesis” or alternative surface model, on a node by node basis, when incoming soundings are not statistically consistent with previous nodal estimates 5. Run the “disambiguation engine” 6. CUBE output = the nodal point-set plus four surfaces 7. Caution One: Rinse, spit and repeat as necessary 8. Caution Two: Consider hydrographic safety (aka Algorithmic Hydrography don’t know Jack!)

14. Step 1 – create a grid  Establish a network of “nodes” (depth estimation points) over the survey area Select nodes to be: Select nodes to be: 1.dense enough to completely capture all features of interest 2.no denser than needed 3.regularly-spaced (or perhaps quad tree) 4.at horizontal locations that are “perfect” since they are selected by convention, and do not involve any measurements

15. Appropriate Resolution Slide credit: Brian Calder

16. Estimation at a Point Estimation at a point means: - Only one true depth - No horizontal uncertainty - Soundings should agree on depth Implications: - Only need to estimate a constant - Must propagate information - Uncertainty gives measure of expected consistency Slide credit: Brian Calder

17. Step 2 – initialize the grid  At each node, maintain sequentially-determined estimates of depth, and depth uncertainty (variance) Need a starting point Need a starting point Could be derived from digitized historic field sheetsCould be derived from digitized historic field sheets Normally, start with depth=0, uncertainty = 1000 mNormally, start with depth=0, uncertainty = 1000 m current estimate can be updated by new local sounding measurements current estimate can be updated by new local sounding measurements means that, in principle, CUBE can be used as a real- time estimation tool (built into the MBES data acquisition software), rather than only in post- processing software like CARIS HIPS means that, in principle, CUBE can be used as a real- time estimation tool (built into the MBES data acquisition software), rather than only in post- processing software like CARIS HIPS

18. Step 3 – start loading soundings  Compare incoming sounding measurements against these existing nodal estimates Each incoming sounding is considered to contain some (imperfect) information about the depth at each of the nodes surrounding it Each incoming sounding is considered to contain some (imperfect) information about the depth at each of the nodes surrounding it propagation strategy involves three processes that appropriately dilute the 3D sounding uncertainty based on the sounding / node separation propagation strategy involves three processes that appropriately dilute the 3D sounding uncertainty based on the sounding / node separation 1.give closer soundings and soundings with smaller depth uncertainty higher weight 2.convert the horizontal sounding uncertainty into a vertical uncertainty at the node 3.make a statistical comparison between the incoming sounding (propagated to the node location), and the already-assimilated information at the node

19. Error Model: Uncertainties Roll Yaw Pitch Heave vv Beamwidth Range Sampling v z SVP Tx/Rx GPS VRU time VRU Sonar Attitude Latency Slide credit: Brian Calder

20. Error Model: Vertical Slide credit: Brian Calder

21. Error Model: Horizontal Slide credit: Brian Calder

22. Error Modeling Slide credit: Brian Calder

23. Error Modeling Slide credit: Brian Calder

24. Step 4 – check nodal consistency  Establish a new “hypothesis” or alternative surface model, on a node by node basis, when incoming soundings are not statistically consistent with previous nodal estimates Why is this so innovative? Two reasons: Why is this so innovative? Two reasons: a) It differs from previous approaches by deferring final judgment on appropriateness of a sounding until all the evidence is in b) It recognizes the possibility that soundings in the same area may be sensing more than one depth trend (e.g. the top of a school of fish as well as the bathymetry beneath them).

25. Robustness: Multiple Hypotheses  Robustness is a: “judicious and grudging elaboration of the model to ensure against particular hazards” (G.E.P. Box).  Observed problems: Spontaneous outliers (e.g., missolved beams) Spontaneous outliers (e.g., missolved beams) Burst mode sounder failures Burst mode sounder failures Systematic artifacts (e.g., dynamic draft) Systematic artifacts (e.g., dynamic draft)  We track multiple potential depths per node: Piecewise defined Dynamic Linear Model (DLM) Piecewise defined Dynamic Linear Model (DLM) Incorporates failures in separate hypotheses Incorporates failures in separate hypotheses Extends ‘memory range’ of estimator Extends ‘memory range’ of estimator Slide credit: Brian Calder

26. CUBE: Multiple Hypotheses Slide credit: Brian Calder

27. Step 5 – resolve inconsistencies  Run the “disambiguation engine” decide which of several hypotheses that may exist at a node is most likely to be correct decide which of several hypotheses that may exist at a node is most likely to be correct Subject to ongoing refinement, engine uses three “metrics” to resolve most probable hypothesis: Subject to ongoing refinement, engine uses three “metrics” to resolve most probable hypothesis: a) The popularity contest. Which hypothesis is consistent with the greatest number of incoming soundings at that node? b) The local consistency test. Which hypothesis is closest to a prediction based on nearest neighbor nodes that have only one hypothesis? c) The external consistency test. Which hypothesis is closest to an “external” (probably lower resolution) reference surface. This surface could be generated by a median filter through all the soundings being processed, or based on an older chart or survey data.

28. CUBE Data Assimilation Slide credit: Brian Calder

29. Step 6 – take a look  CUBE output = the nodal point-set plus four surfaces: 1. Depth – the final depth value at each node, after the competing hypotheses have been disambiguated 2. Uncertainty – the final uncertainty associated each nodal depth 3. The number of hypotheses at each node 4. The uncertainty associated with the selection of the correct hypothesis by the disambiguation engine. This “hypothesis strength”, is a measure of algorithmic certainty - how convinced CUBE is that it's telling you the real story.

30. Output Surface Data Depth Estimate Num. Hypotheses Uncertainty Estimate Hypothesis Strength Slide credit: Brian Calder

31. Step 7 – iterate  Caution One: Rinse, spit and repeat as necessary Axiom 4 - CUBE is a triage tool Axiom 4 - CUBE is a triage tool CUBE does its best to get as much right as possible, but always needs a user at the end to resolve problems that the algorithm cannot CUBE does its best to get as much right as possible, but always needs a user at the end to resolve problems that the algorithm cannot makes the user's job different: not so much about killing dots, more about quality assurance makes the user's job different: not so much about killing dots, more about quality assurance Implies adaptive sampling if used in real-time Implies adaptive sampling if used in real-time

32. CUBE: Per-Day Pass 1 Slide credit: Brian Calder

33. Step 8 – retain human control  Caution Two: Consider hydrographic safety (aka Algorithmic Hydrography don’t know Jack!) Axiom 7 - CUBE inhabits the model world, not the real world Axiom 7 - CUBE inhabits the model world, not the real world Users should not take raw CUBE grids and use them for charting Users should not take raw CUBE grids and use them for charting hydrographer should be allowed to override the algorithm, and retain those (rare) "critical to navigation" sounding values in any given survey hydrographer should be allowed to override the algorithm, and retain those (rare) "critical to navigation" sounding values in any given survey And that leads us to … And that leads us to …

34. CUBE: Cumulative Pass 2 Slide credit: Brian Calder

35. Total Time Comparison Traditional Process: 305.06 hrs. CUBE Process: 124.84 hrs. Slide credit: Brian Calder

36. Total Time Comparison Slide credit: Brian Calder

37. The Navigation Surface  Redefining the Field Sheet  Creation of a database  A process to create multiple products from a single source  Comparison of traditional field sheet to CUBE and Navigation Surface output product.

38. Redefining Field Sheets  Field sheet is typical primary archive of a hydrographic survey  But: It represents the data at only one scale … It represents the data at only one scale … … and not at the best achievable resolution … and not at the best achievable resolution  And, it’s not adequate: For other derived products (e.g., marine geology, habitat studies) For other derived products (e.g., marine geology, habitat studies) For automatic cartographic processing (e.g., automatic contour generation) For automatic cartographic processing (e.g., automatic contour generation) Slide credit: Brian Calder and Shep Smith

39. A possible alternative  Replace the sheet with a database: ‘Best Available’ representation of the sea-floor as a (possibly irregular) grid ‘Best Available’ representation of the sea-floor as a (possibly irregular) grid Attribute each grid node with an estimate of the measurement uncertainty Attribute each grid node with an estimate of the measurement uncertainty  Generate products directly from the database: On demand automatic processing On demand automatic processing Generalize to required resolution from grid with techniques to ensure hydrographic safety Generalize to required resolution from grid with techniques to ensure hydrographic safety Construct other components from generalized grid (e.g., selected soundings or contours) Construct other components from generalized grid (e.g., selected soundings or contours) Slide credit: Brian Calder and Shep Smith

40. Proposed Process Source Soundings Nautical Charts Survey ProductsDatabase Navigation Surface Hydro Vector Database ENC Other Slide credit: Brian Calder and Shep Smith

41. Proposed Process  A Navigation Surface is: A ‘best resolution’ regular or irregular grid of depth estimates, plus uncertainty associated with the estimate A ‘best resolution’ regular or irregular grid of depth estimates, plus uncertainty associated with the estimate A methodology for generalization of the grid to any scale with hydrographic safety A methodology for generalization of the grid to any scale with hydrographic safety  Processing Goals: Speed up interactive time in MBES processing Speed up interactive time in MBES processing Speed up down-stream processes with automatic cartographic methods Speed up down-stream processes with automatic cartographic methods Maintain survey in digital database format for ease of use and maintenance Maintain survey in digital database format for ease of use and maintenance Slide credit: Brian Calder and Shep Smith

42. Snow Passage, AK Slide credit: Brian Calder and Shep Smith

43. Proposed Process Source Soundings Nautical Charts Survey ProductsDatabase Navigation Surface Hydro Vector Database ENC Other Slide credit: Brian Calder and Shep Smith

44. Grid Manipulation Slide credit: Brian Calder and Shep Smith

45. Generalize to Scale of Product  Detail must be sacrificed for clarity of a particular product at a particular scale Multiple products may be defined – don’t want to have to fix at any particular survey scale in the database Multiple products may be defined – don’t want to have to fix at any particular survey scale in the database Significant features have to be preserved, e.g. shoal depths Significant features have to be preserved, e.g. shoal depths  Current Process: Subsampling (binning & suppressing soundings) to maximum resolution (based on survey instructions) Subsampling (binning & suppressing soundings) to maximum resolution (based on survey instructions) Smoothing Linework Smoothing Linework Combination of small features Combination of small features  Defocussing Process: Maintain database at highest resolution Maintain database at highest resolution Manipulate the model, not the cartographic objects Manipulate the model, not the cartographic objects Keep relevant details (shoals, etc.) while simplifying data to scale of final product Keep relevant details (shoals, etc.) while simplifying data to scale of final product Slide credit: Brian Calder and Shep Smith

46. Cartographic Objects from Defocused Surface Slide credit: Brian Calder and Shep Smith

47. Smooth Sheet Soundings Slide credit: Brian Calder and Shep Smith

48. Full Surface Multi-Resolution Slide credit: Brian Calder and Shep Smith

49. Conclusions  CUBE can achieve equivalent performance Does not resolve systematic errors, but does indicate that they exist Does not resolve systematic errors, but does indicate that they exist Computes good estimates even in severe noise Computes good estimates even in severe noise  Operator interaction time is heavily reduced Two runs (and one setup) are required Two runs (and one setup) are required Final time reduction appears to be between 10-20:1 Final time reduction appears to be between 10-20:1 Operator’s task is primarily data quality control rather than editing each outlier Operator’s task is primarily data quality control rather than editing each outlier  Resolution is an essential issue Current implementation uses masks and multiple MapSheets to implement a crude multi-resolution Current implementation uses masks and multiple MapSheets to implement a crude multi-resolution Production implementation will have a more sophisticated solution, probably quad-tree based Production implementation will have a more sophisticated solution, probably quad-tree based Slide credit: Brian Calder and Shep Smith