A Framework of Ontology-Based Knowledge Information Processing for Change Detection in Remote Sensing Data Shutaro Hashimoto 1, Takeo Tadono 1,2, Masahiko Onosato 1, Masahiro Hori 1,2, and Takashi Moriyama 1,2 1 Graduate School of Information Science and Technology, Hokkaido University 2 Earth Observation Research Center, Japan Aerospace Exploration Agency July 28, 2011IGARSS 2011 TH4.T09.21

Background July 28, 2011IGARSS 2011 TH4.T09.22 Needs for automatic image interpretation – especially change detection – to handle large amount of data Mudslides Floods ? Humanlike interpretation requires: – high cognitive ability – versatility

July 28, 2011IGARSS 2011 TH4.T09.23 Solution Emulating manual interpretation using knowledge information processing We propose a framework for change detection – using ontology-based knowledge to recognize and understand targets – input data: optical multispectral data Knowledge Mudslide

July 28, 2011IGARSS 2011 TH4.T09.24 Framework for Change Detection Day 1 Satellite Data Inference Results Day 2 Auxiliary Data e.g. DSM Bayesian Network Query for Target e.g. “mudslide” Information Extraction Inference Analysis of Target Pixel-Based/ Object-Based Image Analysis Knowledge Based on Ontology Bayesian Inference Evidences Synthesis of Knowledge

July 28, 2011IGARSS 2011 TH4.T09.25 Requirements for Knowledge Representation “Vegetation has high NDVI values” “Roads are long and narrow” “Buildings are usually located along Road” “Artificial Forests are often located along River” “Mountains are often covered by Forest” Knowledge representation requires: uncertainty modularity and scalability implicit structural definition of concepts Knowledge Based on Ontology

Remote Sensing Ontology July 28, 2011IGARSS 2011 TH4.T09.26 Heavyweight ontology in remote sensing – 420 concepts Definition Structures Inheritance (B is-a A) Slot (B part-of / attribute-of A) A B soil p/o 1.. leaf chlorophyllcomponent substance Any water chlorophyll a/o 1 p/o 1.. density cluster Anycomponent river field slope continuant entity p/o 1.. a/o 1 geographical object component structurestructural attr. Any contextual change p/o 1.. subeventchange event p/o 0.. Anybefore Anyafter superficial change p/o 1.. componentAny geographical feature p/o 1.. soil layer soilcomponent p/o 1.. water layer watercomponent wood p/o 1.. trunk woodcomponent p/o 1.. tree leafcomponent trunkcomponent sea mountain occurrent p/o 1 soil appearance soil layerafter water appearance p/o 1 water layerafter a/o 1 p/o 1 location slope mudslide subeventsoil appearance substrate p/o 1.. forest treecomponent change event Main Categories p/o 1 Slot 1B Slot 2C a/o 1 A

July 28, 2011IGARSS 2011 TH4.T09.27 Knowledge Based on Ontology Describing relations among some concepts Using Bayesian probability to express uncertainty (1) Concept-Slot Relation (2) Concept- Evidence Relation (3) Co-Occurrence 2 Concepts3 Concepts B C A

(2) July 28, 2011IGARSS 2011 TH4.T09.28 Analysis of Target & Synthesis of Knowledge p/o 0 p/o 1 soil appearance mudslide soil layer before a/o 1 location slope p/o 1 soil layer after p/o 1 subevent soil component soilcomponent Ontology slope angle slope soil layer soil soil layer soil appearance soil layer soil appearance mudslide slope hue soil saturation soil value soil NDVI soil Knowledge Bayesian Network (1) Day 2 soil appearance mudslide slope angle slope Day 1Auxiliary Data soil layer soil hue value NDVI saturation soil layer soil hue value NDVI saturation (3)

July 28, 2011IGARSS 2011 TH4.T09.29 Change Detection Soil Layer Image Object Soil Hue Value Saturation NDVI Satellite Image Day 2 soil appearance mudslide slope angle slope Day 1 Auxiliary Data soil layer soil hue value NDVI saturation soil layer soil hue value NDVI saturation Soil Layer Soil Appearance Day 2 Day 1 Calculate posterior probability of target using Bayesian network Inference of Substance Inference of Object Inference of Change

July 28, 2011IGARSS 2011 TH4.T Experiment  To validate cognitive ability & versatility  Applying to three cases of practical change detection without tuning  Bi-temporal data observed by AVNIR-2 onboard ALOS  3 visible + 1 near-infrared  10 m spatial resolution applied image registration with geometric errors of less than 0.5 pixel

July 28, 2011IGARSS 2011 TH4.T Case 1: Detection of Mudslides in Yamaguchi City, Japan Day 1 (14 June, 2009) Day 2 (30 July, 2009) ©JAXA Mudslides caused by heavy rain in July, 2009

July 28, 2011IGARSS 2011 TH4.T Case 1: Detection of Mudslides in Yamaguchi City, Japan - Inference Results - Day 1 (14 June, 2009) Day 2 (30 July, 2009) ©JAXA soil on day 1 soil appearance mudslide slope Definition of mudslide soil on day 2 p/o 0 p/o 1 soil appearance mudslide soil layer before a/o 1 location slope p/o 1 soil layer after p/o 1 subevent soil component soilcomponent

July 28, 2011IGARSS 2011 TH4.T Case 1: Detection of Mudslides in Yamaguchi City, Japan - Comparison with Human’s Result - small changes => more sensitive than human’s result changes in the flat area => our definition of mudslide doesn’t include changes in flat area

July 28, 2011IGARSS 2011 TH4.T Case 1: Mudslide Detection in Yamaguchi City, Japan - Comparison with Survey Data - Comparison to survey data Results Mudslide No- Mudslide Referen ce Mudslide97 points13 points No- Mudslide 10 points Mudslides in Our Result Collapsed Slopes Mudflow Traces Debris in Survey Data (investigated by Yamaguchi Pref.)

July 28, 2011IGARSS 2011 TH4.T Case 2: Detection of Flooded Areas in Myanmar ©JAXA Day 1 (4 May, 2008) Day 2 (19 June, 2008) Water (Day 1) Water Disappearance Water (Day 2) Floods caused by Cyclone in 2-3 May, 2008

July 28, 2011IGARSS 2011 TH4.T Case 2: Detection of Flooded Areas in Myanmar - Comparison with Human’s Result - Our result not correctly detected due to the existence of clouds Human’s result misdetected edges of clouds

July 28, 2011IGARSS 2011 TH4.T Case 3: Detection of Flooded Areas in Pakistan ©JAXA Day 1 (14 Oct., 2009) Day 2 (1 Sept., 2010) Water (Day 1) Water AppearanceWater (Day 2) Floods caused by heavy rain since late July, 2010

July 28, 2011IGARSS 2011 TH4.T Discussion About 90% accuracy in mudslide detection Our results were better than human’s results  due to using knowledge specialized on targets Fairly good results in all cases without tuning  due to analyzing essential characteristics of each targets using heavyweight ontology Possible to understand and recognize targets as humans do using rich ontology-based knowledge

July 28, 2011IGARSS 2011 TH4.T Conclusions We proposed a framework for change detection – using ontology-based knowledge to recognize and understand targets The experiment showed: – accuracy was about 90 % in mudslide detection – results were better than human’s results without tuning More improvements are ongoing – to extract various information from data, such as spatial information – to describe more expressive knowledge

July 28, 2011IGARSS 2011 TH4.T Thank you!

July 28, 2011IGARSS 2011 TH4.T Probabilistic Inference with Bayesian Network Node: random variable Arc: probabilistic relation ・・・ (1) ・・・ (3) ・・・ (2)

July 28, 2011IGARSS 2011 TH4.T09.222

July 28, 2011IGARSS 2011 TH4.T