1. 2 nd International Conference on Biomedical Ontology (ICBO’11) Ontology-Based Analysis of Event-Related Potentials Gwen Frishkoff 12, Robert Frank 2, Paea LePendu 3, & Snežana Nikolič 1 1 Psychology & Neuroscience, Georgia State University 2 NeuroInformatics Center (NCBO), University of Oregon 3 National Center for Biomedical Ontology (NCBO), Stanford University http://nemo.nic.uoregon.edu

2.  Why ontology-based analysis?  Linking Data to Knowledge in Human Neuroscience  Ontology-based analysis of ERP data  Data  Information Pipeline for automated (and therefore objective)separation of ERP patterns and extraction of summary metrics for each pattern  Information  Knowledge Ontology to represent metrics in semantically structured way so as to automatically classify & label ERP patterns within and across experiments Overview

3.  Why ontology-based analysis?  Linking Data to Knowledge in Human Neuroscience  Ontology-based analysis of ERP data  Data  Information Pipeline for automated (and therefore objective)separation of brainwave (ERP) patterns and automated extraction of summary metrics, which are output to RDF  Information  Knowledge Ontology to represent data (in RDF) and automatically (and therefore objectively) classify & label ERP patterns within and across experiments Overview

4. “The plural of ‘anecdote’ is not ‘data’.” — Roger Brinner (Economist) Assertion #1: In a scientific domain, the priority should be to capture and track assertions about data. Corollary: To capture complex (and presently ill-defined) patterns in data, we need bottom-up (data-driven) analysis.

5. The plural of ‘data’ is not ‘knowledge’. Assertion #2: To draw meaningful inferences from data, they must be linked to a well-structured knowledge base ( ontology ).

6. Knowledge Meaningful (semantically structured) information Information Semi-structured, Processed & compressed data Data Unstructured set of observations (measurements) +1.24  V -.83  V +3.11  V ONTOLOGY DATA INFORMATION Data mining (i.e., analysis) Knowledge engineering Ontology mining?

7. The plural of ‘data’ is not ‘knowledge’. Assertion #3: Ontology  Semantic Structure. It cannot be automatically extracted from data (or patterns in data). Cf. Searle’s Chinese Room argument… Corollary: To build a valid ontology, we need top-down (knowledge-driven) methods (ala BFO/OBO).

8. Introduction to ERP Domain (I): The Data = Measurements of Scalp EEG EEGs (“brainwaves” or flunctuations in brain electrical potentials) are recorded by placing two or more electrodes on the scalp surface. 256-channel Geodesic Sensor Net ~5,000 ms

9. Introduction to ERP Domain (II): From EEG to Event-Related Potentials (ERP)  ERPs (event-related potentials) are the result of averaging across multiple segments of EEG, time- locking to an event of interest. AVERAGE OVER (LOTS OF) EEG SEGMENTS EEG ERP

10. Introduction to ERP Domain (III): Entities of Interest = ERP Patterns (in Data!) ERP patterns characterized by three types of attributes: (1) TIME  latency of peak positive or peak negative potential (left) (2) SPACE  scalp topography of this potential (right); and (3) FUNCTION  experimental context in which these patterns are characteristically observed (e.g., presentation of visual stimulus) 120 ms

11. Tried and true method for noninvasive brain functional mapping Direct measure neuronal activity Whole-brain measurement (at scalp) Millisecond temporal resolution Portable and inexpensive Important clinical applications (e.g., potential biomarkers for AD, presurgical planning) Recent innovations give new windows into rich, multi-dimensional patterns – Rich spatial info (high-density EEG) – Combined temporal & spectral info (JTF) – Multimodal (EEG/ fMRI/MEG) measures 1 sec What’s great about ERPs …

12. If ERPs are so great…. Why are there so few meaningful applications in biomedicine? And why so few (arguably no) cross-lab meta-analyses?

13. Problem #1: Patterns superposed in space & time LATENT (INFERRED) PATTERNS (THIS IS WHAT WE WANT TO TALK ABOUT) MEASURED DATA (THIS IS WHAT WE ACTUALLY MEASURE/OBSERVE!) Superposition

14. Everyone has one, and nobody likes to use anyone else’s. Problem #2a: Patterns (actually, pattern labels ) are like toothbrushes …

15. 410 ms 450 ms 330 ms Consider a Hypothetical Database Query: Show me all the N400 patterns in the database. Peak latency 410 ms “CANONICAL N400” Will the “real” N400 please step forward? Problem #2b: Conversely, different scientists use the same label for incommensurable patterns.

16. Putative “N400”- labeled patterns Parietal N400 ≠ ≠ fN400 Parietal P600 Assertion #3: We cannot ground ERP meta-analysis in prior literature (e.g., text mining). We need a reliable workflow for data analysis & classification.

17. Summary: Motivation for NEMO Lots of different — and equally valid! — methods for pattern analysis Inconsistent and subjective use of metrics and labels for pattern summary and classification No existing methods or tools to support ERP data sharing and integration Assertion #4: The best way to address these issues is to combine data-driven methods for pattern analysis with knowledge-driven methods for ontology development and application (to interpret analysis results)

18. Neural ElectroMagnetic Ontologies  A set of formal (OWL) ontologies for representation of ERP domain concepts  A suite of tools for data-driven extraction and ontology-based annotation of ERP patterns  A database that includes publicly available, annotated data from our NEMO ERP consortium to demonstrate application of ontology for quantitative meta-analysis of results from studies of language and cognition

19.  Why ontology-based analysis?  Linking Data to Knowledge in Human Neuroscience  Ontology-based analysis of ERP data  Data  Information Pipeline for automated (and therefore objective)separation of ERP patterns and extraction of summary metrics for each pattern  Information  Knowledge Ontology to represent data (in RDF) and automatically (and therefore objectively) classify & label ERP patterns within and across experiments Overview

20. Knowledge Meaningful (semantically structured) information Information Semi-structured, Processed & compressed data Data Unstructured set of observations (measurements) FROM DATA TO INFORMATION…. +1.24  V -.83  V +3.11  V Extraction of meaningful patterns (i.e., data analylsis)

21. ERP Pattern Analysis: Current Practice N400 component P3 component “Bumpology” Bumpology^2?

22. NEMO Ontology-based Analysis: Overview 1. ERP Pattern Extraction 2. ERP Metric Extraction 3. RDF Generation (Data Annotation) 4.(Metadata Entry) 5. ERP Pattern Classification

24. 1. NEMO Pattern Extraction NEMO ERP Pattern Extraction Toolkit http://nemoontologies.svn.sourceforge.net/viewvc/nemoontologies/toolkit/release NEMO_ERP_Pattern_Decomposition/ NEMO_ERP_Pattern_Segmentation/

25. Pattern Extraction I: Decomposition Advantages: Data-driven Automated/ Objective Sensitive (able to separate superposed patterns) P100 N100 fP2 P1r/ N3 P1r/ MFN 100ms 170ms 200ms 280ms 400ms Disdvantages: Requires expertise (~vanilla PCA) Not used by majority of ERP researchers NEMO ERP Pattern Extraction Toolkit http://nemoontologies.svn.sourceforge.net/viewvc/nemoontologies/toolkit/release NEMO_ERP_Pattern_Decomposition/

26. Pattern Extraction II: Segmentation NEMO ERP Pattern Extraction Toolkit http://nemoontologies.svn.sourceforge.net/viewvc/nemoontologies/toolkit/release NEMO_ERP_Pattern_Segmentation/

28. 2. Metric Extraction NEMO ERP Metric Extraction Toolkit http://nemoontologies.svn.sourceforge.net/viewvc/nemoontologies/toolkit/release NEMO_ERP_Metric_Extraction/

29. Typical semi-structured representation of ERP data Peak latency measurement (in ms) ERP pattern (extracted from “raw” ERP data using PCA/ICA etc.)

30.  Why ontology-based analysis?  Linking Data to Knowledge in Human Neuroscience  Ontology-based analysis of ERP data  Data  Information Pipeline for automated (and therefore objective)separation of brainwave (ERP) patterns and automated extraction of summary metrics, which are output to RDF  Information  Knowledge Ontology to represent metrics in semantically structured way so as to automatically classify & label ERP patterns within and across experiments Overview

31. Knowledge Meaningful (semantically structured) information Information Semi-structured, Processed & compressed data Data Unstructured set of observations (measurements)ONTOLOGY FROM INFORMATION TO KNOWLEDGE….

32. NEMO Ontology-based Analysis: Overview 1. ERP Pattern Extraction 2. ERP Metric Extraction 3. RDF Generation (Data Annotation) 4. (Metadata Entry) 5. ERP Pattern Classification

33. Recall: Entities of interest (at Stage 1) = Patterns in Data 1 sec TIME SPACE FUNCTION  Modulation of pattern features (time, space, amplitude) in different experiment conditions

34. NEMO Ontology (in a nutshell) L1 : Brain Physiological processes (BFO/OPB) L3 : Brain Physiological data (OBI/IAO)

36. 3. RDF Generation NEMO ERP Metric Extraction Toolkit http://nemoontologies.svn.sourceforge.net/viewvc/nemoontologies/toolkit/release NEMO_ERP_Metric_Extraction/ # OWL Ontology Declaration / Import: GAF-LP1_NN_ERP_data. # Instance Declaration 000: GAF-LP1_NN_ERP_data.

37. Data annotation using RDF “Triples” In natural language = The data represented in cell Z (row A, column 1) is an instance of (“is a”) a peak latency temporal measurement (i.e., the time at which the pattern is of maximal amplitude) Note that the predicate links an instance to a class within NEMO ontology. In RDF form: Subject – Predicate –Object

38. GOAL: Represent extracted information with rich, formal semantics that allow us to reason over data (both within and across datasets) RDF Graph (“triples”)

39. ERP PATTERN CLASSIFICATION

40. 5. Pattern Classification (I) (1) Temporal Criterion (3) Functional Criterion (2) Spatial Criterion

41. 5. Pattern Classification (II) RDF Data loads NEMO ontology RDF Data is opened in Protégé ontology editing software

42. 5. Pattern Classification (III) HermiT Reasoner is used to generate inferences

43. 5. Pattern Classification (IV) Instance-level information (i.e., ERP pattern instances) are successfully classified!

44. Take-home messages 1.For some biomedical applications it may important to capture L3 (DATA) as well as L1 (REALITY) explicitly, i.e., within the ontology 2. In linking the data to the ontology (e.g., for classification/labeling of patterns), it may be important consider data-driven methods for pattern analysis and metric extraction 3. An advantage of this approach is that we can generate relatively stable (non-controversial) representation of data (RDF artifacts), which we will archive and maintain — separate from, but linked to, the ontology — even as the ontology is uncertain & changing. 4. Further, robust representation of data across studies provides basis for valid quantitative meta-analysis, which provide high- quality evidence to inform pattern rules in the ontology

45. Ongoing Work & Open Issues Evolving pattern rules to represent more complex functional criteria (i.e., expt metadata) Temporal reasoning (can we squeeze this into DL/OWL?) Representing uncertainty in pattern rules & classification of pattern instances (beyond Evidence Codes?) Clinical applications: Pilot cross-lab work with aphasics (stroke & TBI patients with language disorders)

46. Funding from the National Institutes of Health (NIBIB), R01-MH084812 (Dou, Frishkoff, Malony) NEMO Ontology Task Force Robert M. Frank (NIC) Dejing Dou (CIS) Paea LePendu (CIS) Haishan Liu (CIS) Allen Malony (NIC, CIS) Jason Sydes (CIS) *Snezana Nikolic (PSY, GSU) *emeritus Acknowledgments www.nemo.nic.uoregon.edu NEMO EEG/MEG Data Consortium Tim Curran (U. Colorado) Dennis Molfese (U. Louisville) John Connolly (McMaster U.) Kerry Kilborn (Glasgow U.) Charles Perfetti (U. Pittsburgh) Special thanks to: Maryann Martone & associates (NIF) Jessica Turner (cogPO) Angela Laird (BrainMap) Sivaram Arabandi (OGMS) YOU (BIO-ONTOLOGY COMMUNITY)

47. Recent References Frishkoff, G., Frank, R., LePendu, P., & Nikolic, S. (2011, in press). Ontology- based Analysis of Event-Related Potentials. Proceedings of the International Conference on Biomedical Ontology (ICBO'11). Frishkoff, G., Frank, R., Sydes, J., Mueller, K., & Malony, A. (2011, subm). Minimal Information for Neural Electromagnetic Ontologies (MI-NEMO): A standards-compliant workflow for analysis and integration of human EEG. Standards in Genomic Sciences (SIGS). Liu, H., Frishkoff, G., Frank, R. M. F., & Dou, D. (2011, subm). Integration of Human Brain Data: Metric and Pattern Matching across Heterogeneous ERP Datasets. Journal of Neurocomputing. Frank, D. & Frishkoff, G. A. (2011, in prep.). The NEMO ERP Analysis Toolkit: Combining data-driven and knowledge-driven methods for ERP pattern analysis. Neuroinformatics. Frishkoff, G.A., Dou, D., Frank, R., LePendu, P., and Liu, H. (2009). Development of Neural Electromagnetic Ontologies (NEMO): Representation and integration of event-related brain potentials. Proceedings of the International Conference on Biomedical Ontologies (ICBO09). July 24-26, 2009. Buffalo, NY.

48. Annotating Data in RDF Data Annotation – The process of marking up or “tagging” data with meaningful symbols; tags may come from ontology  linked to a URI URI (Uniform Resource Identifier) – A compact sequence of characters that identifies an abstract or physical resource (typically located on the Web) RDF (Resource Description Framework) – RDF is a directed, labeled graph (data model) for representing information (typically on the Web) *See Glossary (http://www.seiservices.com/nida/1014080/ReadingRoom.aspx)

49. The “RDF Triple” In RDF form: Subject – Predicate –Object In natural language: The data represented in row A is an instance of (“is a”) some ERP pattern. That is, measurements (cells) are “about” ERP patterns (rows). In graph form:

50. RDF Triple #2 In natural language = The data represented in cell Z (row A, column 1) is an instance of (“is a”) a peak latency temporal measurement (i.e., the time at which the pattern is of maximal amplitude) In RDF form: Subject – Predicate –Object

51. RDF Triple #3 This graph represents an assertion, expressed in RDF = The data represented in cell Z is a temporal property of the ERP pattern represented in row A

52. NOTE: Pattern definition is encoded in the ontology (not in RDF data rep!)

53. This is the inference that we want to make

54. Pattern classification is the goal

55.  The challenge (EEG pattern classification)  The methods & tools  ontologies  RDF database  Proof of concept (a worked example) Case Study 5 (NEMO): Neural ElectroMagnetic Ontologies

56. RECALL: Pattern definition is encoded in the ontology (not in RDF data rep!) HOW?

57. The rule (just the temporal criterion)as it appears in Protégé Protégé rendering OWL/RDF rendering

58. RECALL: The goal is to formulate pattern definitions, use them to classify data, and ultimately to revise them based on meta-analysis results Observed Pattern = “N400” iff  Event type is onset of meaningful stimulus (e.g., word) AND  Peak latency is between 300 and 500 ms AND  Scalp region of interest (ROI) is centroparietal AND  Polarity over ROI is negative(>0)

59. Linking data to ontology — Step 1: A worked example (Formulating rule) First, we write the ERP pattern rule as follows: IF (1) 001 type ERP_spatiotemporal pattern and (2) 002 type peak_latency_measurement_datum and (3) 002 is_peak_latency_measurement_of 001, and (4) 002 has_numeric_value X, and (5) 500 >= X >= 300 (X has datatype decimal) (in reality, there are spatial, temporal, & functional criteria…) THEN (6) 001 type N400_pattern

60. Next, we convert the rule to a SPARQL query by replacing natural language terms with corresponding URI (tags) from NEMO ontology type  rdf:type ERP_spatiotemporal_pattern  NEMO_0000093 peak_latency_measurement  NEMO_0745000 is_measurement_of  NEMO_9278000 has_numeric_value  NEMO_7943000 Linking data to ontology — Step 2: Formulating rule as SPARQL query

61. Finally, we load Virtuoso’s SPARQL interface http://nemo.nic.uoregon.edu:8890/sparql & then cut and paste the query into the Query textbox and click Run Query. …. And Virtuoso returns the following results (for ex): As a result, we can deduce that ERP observations 0002, 0003, 0004, & 0006 are N400 pattern instances… QED Linking data to ontology — Step 3: Executing the SPARQL query

62. Cycles of Knowledge discovery & Knowledge Engineering (i.e., Onto Dev’t)

63. Linking Shared Data & Resources (http://linkeddata.org/) NIF (Neurolex) NIF (Neurolex) NEMO CARMEN (MINI) CARMEN (MINI) HeadIT CogPO

64. NOW YOU SHOULD KNOW… What is an ontology & what’s it for? – Why bother? – What are some “best practices” in ontology design & implementation? What is RDF & what’s it for? – How does RDF represent information? – How is it used to link data to ontologies? – How can ontology-based annotation be used to support classification of data?