Developing Visualization Techniques for Semantics-based Information Networks Rich Keller David Hall NASA Ames QSS Group, Inc. Information Sharing and Integration Group Computational Sciences Division NASA Ames Research Center Moffett Field, CA Virtual Iron Bird Workshop, Monterey, CA April 2, 2004

Goals of our Work Goal #2 (Engineering): Develop an effective visual interface for an existing NASA information / knowledge management tool Goal #1 (Scientific): Understand how semantic knowledge can be exploited to help visualize large network- structured information spaces (… like the Semantic Web)

Talk Outline SemanticOrganizer System Visualization Problem Proposed Semantic Approaches to Visualization Problem Work in progress!

What is SemanticOrganizer? A semantics-based shared information space: designed to support distributed science and engineering project teams Facilitates information sharing, integration, correlation and dependency tracking Core is a digital information repository: users upload & download heterogeneous information (images, datasets, documents, and various types of scientific/engineering records) Features semantic cross-linkage: enables rapid intuitive access to interrelated information; permits linking facts and evidential information to scientific/engineering conclusions Serves as organizational memory: preserves details of investigative fieldwork, labwork, & associated data collection/ data analysis activities and processes

Operational Status First deployed in 2001 Over 500 registered individual users from over 50 organizations within NASA Over 50 projects hosted Over 45,000 information nodes & 150,000 links in repository Over 14,000 electronic files stored (documents, image, datasets) Over 12,000 archived messages as of 4/1/04

Application Features SemanticOrganizer Applications MARTE Mars Analog Mission Moffett Airshow Investigation MER Hypothesis Tracking Mobile Agents Mars Exploration ScienceOrganizer Real-time equipment control Automated experimentation Both Collaborative image annotation Microsoft Office macros lists & archive InvestigationOrganizer Fault tree viewer Event sequence editor …

How is the Information Repository structured? person photo measurement site instrument sample document Links: defined relationships among resources Attached files: electronic products associated with resources (in almost any type of file format) Attributes: properties of resources (metadata) Nodes: key science or engineering resources (describing people, places, systems, hypotheses, evidence) date size format Ontology: Specifies the types of nodes, attributes and links defined for each different application (RDFS-type Representation) Rules: Add/modify nodes, links & attributes in the network

DNA sequence image field trip culture person sample photographic image SEM image Scientific Investigation Ontology (partial) other experiment Scientific Information Resouces project measurement field site equipment camera gas chromatograph stub O2 microsensor N2 microsensor SEM O2 concentration N2 concentration spectrometer spectrograph chromatogram other micrograph cultivated-from cultivated-by has-genetic-sequence pictured-in researcher lab tech

May-June 2001 Field Trip Baja Study Area In Situ Diel Experiment Measurements 7 Images 99 Documents 3 EMERG Project People 15 Samples 19 Brad’s Trip Planning Document a c d f gh i j a: has logistics b: samples collected c: has objectives d: conducted experiment e: located at f: trip participants g: destination h: trip for i: measurements taken j: has photodocumentation k: site for l: collected at m: experiment site for n: experiment staff o: has custodian p: pictured in q: has sequence info r: source of s: has measurement Links b Strawman for Focus group Document aa t: employed in u: collected by v: led by w: authored by x: has subexperiments y: has measurement pass z: generated measurements aa: associated documents bb: has test point Example Semantic Information Space Pond 4 near 5 Field Site Projects 2 Samples 56 e k l 7 Experiments 6 m t x ybb Greenhouse Sulfate Manipulation Experiment Test Point 18 Experiments 3 O 2 Measurement 36 z Thermal Variance Experiment Experiments 2 x Salinity Experiment Diel Cycle Experiment Measurement Pass 2 bb s s People 5 n v M Measurement SC-8-11 Culture 16S3 rRNA DNA Sequence Bebout, Brad Scientist Carpenter, Steve Lab Technician o r q s q u w P4Mat-16 Mat Sample Images 33 p 8 Instance space

Current SemanticOrganizer Interface Links to Related Records create new records modify record icon identifies record type search for records Right side displays metadata for the current repository record being inspected Left side uses semantic links to display all information related to the repository record shown on the right semantic links related records (click to navigate) current record

Interface Problems Graphical overview of information space needed for: –Comprehension of information scope and context –Non-local navigation Can’t display entire information space –Over 45,000 nodes –Over 150,000 links Can’t make sense of entire information space

Remedy: Filtering and Abstraction Filtering: Remove nodes/links Abstraction: Replace a set of nodes/links with a smaller number of nodes/links Q: What is the basis for filtering or abstraction?

Sources of Knowledge for Filtering/Abstraction Graph-theoretic: based on topological properties of network (e.g. cut points) Content-based: using textual content stored in nodes (e.g., as in Web page clustering) Semantics-based: – ontology (node-type, link-type, subsumption) – auxiliary information: importance/intrinsicality of nodes/links usage context

Semantic Approaches to Simplifying Information Space Presentation 1.Contextual Filtering 2.Semantic Structure Abstraction 3.Semantic Navigation

1. Contextual Filtering Observation: Not all nodes/links are relevant in a given context Proposed Approach: Define explicit constraints that generate a meaningful subgraph of nodes in a specific context Context examples: a specific scientific field trip a specific project a specific location (e.g., a scientific laboratory or field site)

Example: Using Constraints to define a Field Trip Context FieldTripContext(f) = { {f}  S  M  SA  P  R  E  FS  I } where: FieldTrip(f) // f is a node of type FieldTrip S = {s | Sample(s) ∧ SamplesCollected(f, s)} M = {m | Measurement(m) ∧ MeasurementTaken(f, m)} SA = {sa | StudyArea(sa) ∧ Destination(f, sa)} P = {p | Person(p) ∧ TripParticipant(f, p)} R = {r | Project(r) ∧ TripFor(f, r)} E = {e | Experiment(e) ∧ ConductedExperiment(f, e)} FS ={fs | FieldSite(fs) ∧ CollectedAt(fs, s) ∧ s  S} I = {i | Image(i) ∧ PicturedIn(s, i) ∧ s  S} a) subset of nodes linked directly to a field trip node + b) images of samples gathered during that trip and field sites where those samples were collected Field Trip Context = a b

Samples 19 May-June 2001 Field Trip Baja Study Area In Situ Diel Experiment Measurements 7 EMERG Project People 15 P4Mat-16 Mat Sample Images 33 M Measurement Pond 4 near 5 Field Site a d f gh i b Results of Applying Field Trip Filter p s e Projects 2 Experiments 6 m Samples 56 aa c j Images 99 Documents 3 Brad’s Trip Planning Document Strawman for Focus group Document 8 u SC-8-11 Culture 16S3 rRNA DNA Sequence Bebout, Brad Scientist Carpenter, Steve Lab Technician o r q q t x ybb Greenhouse Sulfate Manipulation Experiment Test Point 18 Experiments 3 O 2 Measurement 36 z Thermal Variance Experiment Experiments 2 x Salinity Experiment Diel Cycle Experiment Measurement Pass 2 bb s People 5 n v 8 w k l 7

2. Semantic Structure Abstraction Proposed Approach: apply semantic patterns to identify these substructures represent them as abstract nodes display them using familiar representation Observation: Graphs can obscure structure! Certain graph substructures are better depicted using more familiar visual representations Hierarchical structures  trees List structures  arrays Cross-correlated structures  tables Time sequences  PERT charts

Semantic Structure Abstraction: Approach x y bb Greenhouse Sulfate Manipulation Experiment Test Point 5 Experiments 3 O 2 Measurement 10 Thermal Variance Experiment Experiments 2 x Salinity Experiment Diel Cycle Experiment Measurement Pass 2 bb s Gas Flux Experiment Peak Cycle Experiment 1. Recognize patterns Greenhouse Sulfate Manipulation Experiment Test Point x Measurement x Pass 2. Represent as abstract nodes Greenhouse Sulfate Manipulation SalinityThermal Variance Diel Cycle Gas Flux Peak cycle 3. Display appropriately BCDE Pass 12 m8m1m2m3m4m5 Test Point A m6m7m9m10 O 2 Measurement { Experiment Hierarchy 2-dimensional measurement indexing structure

3. Semantic Navigation Proposed Approach: Move from current detailed, fine-grained interface to more abstract navigation interface Abstract away the specific links and present only clusters of nodes radiating out from a focal node Use a semantics-based focus+context style display (e.g., fisheye, hyperbolic) Observation: High-level semantic categories in an ontology can help users visualize and navigate the information space in a more effective, rapid, intuitive fashion

More Abstract Interface: Bull’s-Eye Navigator field trip artifacts related to “field trip” (e.g., sample-X) people related to “field trip” artifacts (e.g., lab tech who analyzed sample-X) scientists lab techs traverse expts … samples, msmts … projects orgs … labs sites … (1 link away) (2 links away) Focal Region Context Region Compact representation of information space surrounding focal node docs Semantic categories: People Places Activities Artifacts Social Structures

Summary Large information spaces are difficult to comprehend and navigate Visualization can help Semantic information provides leverage for visualization Three examples: –Contextual Filtering –Semantic Structure Abstraction –Semantic Navigation