1. A Framework of Ontology-Based Knowledge Information Processing for Change Detection in Remote Sensing Data Shutaro Hashimoto 1, Takeo Tadono 1,2, Masahiko Onosato 1, Masahiro Hori 1,2, and Takashi Moriyama 1,2 1 Graduate School of Information Science and Technology, Hokkaido University 2 Earth Observation Research Center, Japan Aerospace Exploration Agency July 28, 2011IGARSS 2011 TH4.T09.21

2. Background July 28, 2011IGARSS 2011 TH4.T09.22 Needs for automatic image interpretation – especially change detection – to handle large amount of data Mudslides Floods ? Humanlike interpretation requires: – high cognitive ability – versatility

3. July 28, 2011IGARSS 2011 TH4.T09.23 Solution Emulating manual interpretation using knowledge information processing We propose a framework for change detection – using ontology-based knowledge to recognize and understand targets – input data: optical multispectral data Knowledge Mudslide

4. July 28, 2011IGARSS 2011 TH4.T09.24 Framework for Change Detection Day 1 Satellite Data Inference Results Day 2 Auxiliary Data e.g. DSM Bayesian Network Query for Target e.g. “mudslide” Information Extraction Inference Analysis of Target Pixel-Based/ Object-Based Image Analysis Knowledge Based on Ontology Bayesian Inference Evidences Synthesis of Knowledge

5. July 28, 2011IGARSS 2011 TH4.T09.25 Requirements for Knowledge Representation “Vegetation has high NDVI values” “Roads are long and narrow” “Buildings are usually located along Road” “Artificial Forests are often located along River” “Mountains are often covered by Forest” Knowledge representation requires: uncertainty modularity and scalability implicit structural definition of concepts Knowledge Based on Ontology

6. Remote Sensing Ontology July 28, 2011IGARSS 2011 TH4.T09.26 Heavyweight ontology in remote sensing – 420 concepts Definition Structures Inheritance (B is-a A) Slot (B part-of / attribute-of A) A B soil p/o 1.. leaf chlorophyllcomponent substance Any water chlorophyll a/o 1 p/o 1.. density cluster Anycomponent river field slope continuant entity p/o 1.. a/o 1 geographical object component structurestructural attr. Any contextual change p/o 1.. subeventchange event p/o 0.. Anybefore Anyafter superficial change p/o 1.. componentAny geographical feature p/o 1.. soil layer soilcomponent p/o 1.. water layer watercomponent wood p/o 1.. trunk woodcomponent p/o 1.. tree leafcomponent trunkcomponent sea mountain occurrent p/o 1 soil appearance soil layerafter water appearance p/o 1 water layerafter a/o 1 p/o 1 location slope mudslide subeventsoil appearance substrate p/o 1.. forest treecomponent change event Main Categories p/o 1 Slot 1B Slot 2C a/o 1 A

7. July 28, 2011IGARSS 2011 TH4.T09.27 Knowledge Based on Ontology Describing relations among some concepts Using Bayesian probability to express uncertainty (1) Concept-Slot Relation (2) Concept- Evidence Relation (3) Co-Occurrence 2 Concepts3 Concepts B C A

8. (2) July 28, 2011IGARSS 2011 TH4.T09.28 Analysis of Target & Synthesis of Knowledge p/o 0 p/o 1 soil appearance mudslide soil layer before a/o 1 location slope p/o 1 soil layer after p/o 1 subevent soil component soilcomponent Ontology slope angle slope soil layer soil soil layer soil appearance soil layer soil appearance mudslide slope hue soil saturation soil value soil NDVI soil Knowledge Bayesian Network (1) Day 2 soil appearance mudslide slope angle slope Day 1Auxiliary Data soil layer soil hue value NDVI saturation soil layer soil hue value NDVI saturation (3)

9. July 28, 2011IGARSS 2011 TH4.T09.29 Change Detection Soil Layer Image Object Soil Hue Value Saturation NDVI Satellite Image Day 2 soil appearance mudslide slope angle slope Day 1 Auxiliary Data soil layer soil hue value NDVI saturation soil layer soil hue value NDVI saturation Soil Layer Soil Appearance Day 2 Day 1 Calculate posterior probability of target using Bayesian network Inference of Substance Inference of Object Inference of Change

10. July 28, 2011IGARSS 2011 TH4.T09.210 Experiment  To validate cognitive ability & versatility  Applying to three cases of practical change detection without tuning  Bi-temporal data observed by AVNIR-2 onboard ALOS  3 visible + 1 near-infrared  10 m spatial resolution applied image registration with geometric errors of less than 0.5 pixel

11. July 28, 2011IGARSS 2011 TH4.T09.211 Case 1: Detection of Mudslides in Yamaguchi City, Japan Day 1 (14 June, 2009) Day 2 (30 July, 2009) ©JAXA Mudslides caused by heavy rain in 19-26 July, 2009

12. July 28, 2011IGARSS 2011 TH4.T09.212 Case 1: Detection of Mudslides in Yamaguchi City, Japan - Inference Results - Day 1 (14 June, 2009) Day 2 (30 July, 2009) ©JAXA soil on day 1 soil appearance mudslide slope Definition of mudslide soil on day 2 p/o 0 p/o 1 soil appearance mudslide soil layer before a/o 1 location slope p/o 1 soil layer after p/o 1 subevent soil component soilcomponent

13. July 28, 2011IGARSS 2011 TH4.T09.213 Case 1: Detection of Mudslides in Yamaguchi City, Japan - Comparison with Human’s Result - small changes => more sensitive than human’s result changes in the flat area => our definition of mudslide doesn’t include changes in flat area

14. July 28, 2011IGARSS 2011 TH4.T09.214 Case 1: Mudslide Detection in Yamaguchi City, Japan - Comparison with Survey Data - Comparison to survey data Results Mudslide No- Mudslide Referen ce Mudslide97 points13 points No- Mudslide 10 points Mudslides in Our Result Collapsed Slopes Mudflow Traces Debris in Survey Data (investigated by Yamaguchi Pref.)

15. July 28, 2011IGARSS 2011 TH4.T09.215 Case 2: Detection of Flooded Areas in Myanmar ©JAXA Day 1 (4 May, 2008) Day 2 (19 June, 2008) Water (Day 1) Water Disappearance Water (Day 2) Floods caused by Cyclone in 2-3 May, 2008

16. July 28, 2011IGARSS 2011 TH4.T09.216 Case 2: Detection of Flooded Areas in Myanmar - Comparison with Human’s Result - Our result not correctly detected due to the existence of clouds Human’s result misdetected edges of clouds

17. July 28, 2011IGARSS 2011 TH4.T09.217 Case 3: Detection of Flooded Areas in Pakistan ©JAXA Day 1 (14 Oct., 2009) Day 2 (1 Sept., 2010) Water (Day 1) Water AppearanceWater (Day 2) Floods caused by heavy rain since late July, 2010

18. July 28, 2011IGARSS 2011 TH4.T09.218 Discussion About 90% accuracy in mudslide detection Our results were better than human’s results  due to using knowledge specialized on targets Fairly good results in all cases without tuning  due to analyzing essential characteristics of each targets using heavyweight ontology Possible to understand and recognize targets as humans do using rich ontology-based knowledge

19. July 28, 2011IGARSS 2011 TH4.T09.219 Conclusions We proposed a framework for change detection – using ontology-based knowledge to recognize and understand targets The experiment showed: – accuracy was about 90 % in mudslide detection – results were better than human’s results without tuning More improvements are ongoing – to extract various information from data, such as spatial information – to describe more expressive knowledge

20. July 28, 2011IGARSS 2011 TH4.T09.220 Thank you!

21. July 28, 2011IGARSS 2011 TH4.T09.221 Probabilistic Inference with Bayesian Network Node: random variable Arc: probabilistic relation ・・・ (1) ・・・ (3) ・・・ (2)

22. July 28, 2011IGARSS 2011 TH4.T09.222

23. July 28, 2011IGARSS 2011 TH4.T09.223 0.30.4 0.5 0.6