Bridget McInnes Ted Pedersen Serguei Pakhomov

Bridget McInnes Ted Pedersen Serguei Pakhomov
11/12/2018 UMLS-Interface and UMLS-Similarity: Open Source Software for Measuring Paths and Semantic Similarity Bridget McInnes Ted Pedersen Serguei Pakhomov

Objective Develop tools to automatically compute the semantic similarity between two concepts in the biomedical domain using measures originally developed for general English using the Unified Medical Language System (UMLS) The objective of this work was to develop tools to automatically computer the semantic similarity between two concepts in the biomedical domain using measures originally developed for general English We obtain these biomedical concepts from the Unified Medical Language System UMLS

Motivation Clustering symptoms and disorders found in the text of clinical reports for post marking medication safety and surveillance Identification of patients for clinical studies Improving the sensitivity of document retrieval of scientific journals and clinical reports Development of terminologies and ontologies Clustering of biomedical documents Word sense disambiguation

Unified Medical Language System
Knowledge representation framework Contains 3 Main components: Metathesaurus Semantic Network SPECIALIST Lexicon The Unified Medical Language System (UMLS) which is a knowledge representation framework consisting of three main components: 1. The Metathesaurus 2. The Semantic Network 3. The Specialist Lexicon Our focus with respect to these packages is the Metathesaurus

Metathesaurus Semi-automatically integrates biomedical concepts from over a 100 controlled medical terminologies Source vocabularies are organized based on their Atomic Unique Identifiers Metathesaurus is organized based on their Concept Unique Identifier (CUI) The Metathesaurus integrates biomedical concepts from over 100 controlled medical terminologies into a single source The concepts from the terminologies or often called source vocabularies are referred as Atomic Unique Identifiers (AUIs). These AUIs are semi-automatically clustered into Concept Unique Identifiers CUIs

Concept Unique Identifiers (CUIs)
CUI C Cold Temperature AUI A Cold Temperature AUI A Low Temperature For example, two source vocabularies in the Metathesaurus are MSH and SNOMEDCT In MSH and SNOMEDCT there exists a concept for “Cold Temperature” these AUIs are combined to form a CUI for the concept Cold Temperature MSH SNOMED-CT

CUI Information The concepts (AUIs) from the source vocabularies may contain information about the concept such as its Definition Relation information between the concepts The information from the AUIs can be obtained through their respective CUIs These concepts from the source vocabularies may be encoded with information about the concept such as its definition or relationship with other concepts This information encoded at the AUI level may be obtained through their respective CUIs

Relations between CUIs in MSH
is-a AUI A Temperature AUI A Cold Temperature CUI C Temperature PAR/CHD (MSH) For example, in MSH there exists an is-a relationship between the concepts Cold Temperature and Temperature This is carried over as a PAR/CHD between the CUIs Cold Temperature and Temperature The Metathesaurus keeps track of where the relationships originated CUI C Cold Temperature MSH

Relations between CUIs in SNOMED-CT
is-a AUI A Temperature AUI A Low Temperature CUI C Temperature PAR/CHD (SNOMED-CT) So for example, If a relationship existed between Low Temperature and Temperature in SNOMED CT – this also would be carried to the CUI level CUI C Cold Temperature SNOMED-CT

Multiple Relations PAR/CHD (SNOMED-CT) CUI C0039476 Temperature
Cold Temperature PAR/CHD (MSH) CUI C Temperature Creating two entries in the Metathesaurus for the same pair of CUIs – one for each source. CUI C Cold Temperature

Relation Information MRHIER MRREL AUI A0123939 Temperature
CUI C Temperature Relation Relation This relation information is stored in two tables in the Metathesaurus. MRHIER which stores the relations between AUIs in their respective sources And MRREL which stores the relations between CUIs. AUI A Cold Temperature CUI C Cold Temperature

MRREL and MRHIER MRHIER MRREL
Contains the full path to root relations between AUIs from each of the sources is-a part-of MRREL Contains the pairwise relations between CUIs Relations: PAR/CHD RB/RN It is possible to generate MRHEIR from MRREL except for the following sources: AIR MSH SNM2 USPMG OMS MRHIER contains the full path to root relations between the AUI for each of the different sources MRREL contains the pairwise relations between the CUIs derived from these sources. These relations are labeled PAR/CHD relations in MRREL MRREL contains other types of relations - the two considered hierarchical in the literature are the PAR/CHD relations and the RB/RN (broader or narrower) It is possible to generate MRHIER from MRREL through the PAR/CHD relations for most sources although this is not the case for all of them. For example, MSH because a MSH concept may have different children depending on their location within the hierarchy. NOTES: don’t read the full path-to-root is a transitive closure of the pairwise PAR/CHD relations which does not hold true for MSH because a MSH concept may have different children depending on their location within the hierarchy

CUI versus AUI Hierarchy
The benefit of using CUIs Ability to obtain the relation information between concepts across sources Ability to obtain the relation information between concepts using more than one type of relation: PAR/CHD – parent/child (relation in MRHIER) RB/RN – narrower/broader SIB – sibling RL – concepts are similar or ‘alike’ The benefit of using AUIs Ability to obtain relation information (PAR/CHD) between concepts in the same source very quickly incorporates tree positional information for sources such as MSH UMLS-Query by Shah and Musen, 2008 There are benefits to using either the AUI hierarchy in MRHIER and the CUI hierarchy in MRREL The benefit of using CUIs is that ability to obtain the path information between in concepts in different sources and over relations outside of just the PAR/CHD relation But the benefit of using AUIS is that the PAR/CHD path information between concepts incorporates the tree positional information for the sources that use this information such as MSH. Currently, there is one package that we know of that obtains path information using MRHIER which is UMLS-Query by Shah and Musen

UMLS-Interface Perl interface to the UMLS present locally in a MySQL database. Its main purpose is to returns path information about CUIs using the relation information in MRREL All possible paths to the root Shortest path between two concepts The UMLS-Interface package is a perl interface to the UMLS which is present locally in a MySQL database Its main purpose is to return path information about CUIs using information from MRREL such as all of the possible paths from the CUI to the root given a specified set of sources and relations

UMLS-Similarity A suite of perl modules that implement a number of path-based semantic similarity measures to determine the similarity between two CUIs in the UMLS Measures are path-based because they rely on the location of the concepts in a hierarchy The path information is obtained using UMLS-Interface Semantic Similarity Measures: Path measure Conceptual Distance (Rada, et. al, 1989) Leacock and Chodorow, 1998 Wu and Palmer, 1994 Nguyen and Al-Mubaid, 2006 UMLS-Similarity package is a Perl package that allows you to take the semantic similarity between two CUIs given a specified set of sources and relations The measure currently implemented in UMLS-Similarity are the path based measures proposed by Leacock and Chodorow Wu and Palmer Nguyen and Al-Mubaid And the conceptual distance measure proposed by Rada, at. Al. and the path measure These measure are called path-based measures because they rely on the location of the concepts in a hierarchy – the path information is obtained from the UMLS-Interface package which returns the path information given a set of specified sources and relations

Semantic Similarity Example
Path measure 1 Sim(c1,c2) = N where N = # links in the shortest path between the two concepts c1 and c2 In the next couple slides I am going to show and example using the path measure which is one over the number of nodes in the shortest path between two concepts

Similarity Given Specified Sources
Body Regions PAR (SNOMED-CT) C Anatomic Part PAR (MSH) So in this example would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using MSH, the similarity is going to be one over one since there is only one link between the two concepts although when using SNOMEDCT the similarity is ½ since there are two links between the two concepts. If we combine the MSH and SNOMEDCT sources the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs PAR (SNOMED-CT)

Body Regions PAR (SNOMED-CT) C Anatomic Part Similarity = 1/1 PAR (MSH) So in this example would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using MSH, the similarity is going to be one over one since there is only one link between the two concepts although when using SNOMEDCT the similarity is ½ since there are two links between the two concepts. If we combine the MSH and SNOMEDCT sources the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs PAR (SNOMED-CT)

Body Regions PAR (SNOMED-CT) Similarity = 1/2 C Anatomic Part PAR (MSH) So in this example would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using MSH, the similarity is going to be one over one since there is only one link between the two concepts although when using SNOMEDCT the similarity is ½ since there are two links between the two concepts. If we combine the MSH and SNOMEDCT sources the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs PAR (SNOMED-CT)

Body Regions PAR (SNOMED-CT) Similarity = 1/1 C Anatomic Part PAR (MSH) So in this example would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using MSH, the similarity is going to be one over one since there is only one link between the two concepts although when using SNOMEDCT the similarity is ½ since there are two links between the two concepts. If we combine the MSH and SNOMEDCT sources the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs PAR (SNOMED-CT)

Similarity Given Specified Relations
Body Regions RB (MSH) C Anatomic Part PAR (MSH) And in this example we would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using PAR/CHD relations, the similarity is going to be one over one since there is only one link between the two concepts although when using the RB/RN the similarity is ½ since there are two links between the two concepts. If we combine the PAR/CHD and RB/RN relations the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs RB (MSH)

Body Regions RB (MSH) C Anatomic Part Similarity = 1/1 PAR (MSH) And in this example we would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using PAR/CHD relations, the similarity is going to be one over one since there is only one link between the two concepts although when using the RB/RN the similarity is ½ since there are two links between the two concepts. If we combine the PAR/CHD and RB/RN relations the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs RB (MSH)

Body Regions Similarity = 1/2 RB (MSH) C Anatomic Part PAR (MSH) And in this example we would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using PAR/CHD relations, the similarity is going to be one over one since there is only one link between the two concepts although when using the RB/RN the similarity is ½ since there are two links between the two concepts. If we combine the PAR/CHD and RB/RN relations the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs RB (MSH)

Body Regions Similarity = 1/1 RB (MSH) C Anatomic Part PAR (MSH) And in this example we would like to calculate the similarity between Limbs and Body Region using the path measure given a set of specified sources Using PAR/CHD relations, the similarity is going to be one over one since there is only one link between the two concepts although when using the RB/RN the similarity is ½ since there are two links between the two concepts. If we combine the PAR/CHD and RB/RN relations the similarity is going to be one over the number of links in the shortest path between the two concept which would be one. C Limbs RB (MSH)

Functional Validation
Comparison with Previous Work: Pedersen, et al. 2007 Nguyen and Al-Mubaid, 2006 Caviedes and Cimino, 2004 We compare the results obtained by UMLS-Similarity to those reported by Pedersen, et. Al. Nguyen and Al-Mubaid and Caviedes and Cimino

Pedersen, et al. Semantic Similarity Measures Source Data
Path Leacock and Chodorow, 1998 Source SNOMEDCT Data 29 medical terms pairs Similarity determined by: 9 Medical Coders 3 Physicians 4 Point Scale 4 – practically synonymous 3 – related 2 – marginally related 1 - unrelated Spearman’s Rank Correlation Coefficient Pedersen evaluate the semantic similarity measure proposed by Leacock and Chodorow and the path measure using SNOMEDCT prior to its inclusion in the UMLS Their dataset consists of 29 medical term pairs whose similarity was determined by 9 Medical Coders and 3 Physicians using a 4 point scale where four indicates that the terms are practically synonymous and 1 indicates that they are unrelated The authors evaluate the measures by determining the correlation between the human annotations and the measures using Spearman’s rank correlation coefficient

Comparison with Pedersen, et al.
Semantic Similarity Measures Path Leacock and Chodorow, 1998 Source: SNOMED-CT from UMLS 2008AB Relations: PAR/CHD Comparison with human annotations Spearman Rank Correlation Coefficient We compare the correlation results obtained by UMLS-Similarity on the same dataset using SNOMEDCT and the PAR/CHD relations in the 2008AB version of the UMLS

Comparison with Pedersen, et al.
Measure Physician Coder path Pedersen, et. al. 0.36 0.51 UMLS-Similarity 0.35 0.50 Leacock and Chodorow The results show that the correlation obtained by UMLS-Similarity and the human annotators when using the path measure is only one percentage point lower than those obtained by Pedersen, et. Al. and the results obtained using the measure proposed by Leacock and Chodorow are identical to those obtained by Pedersen, et. al.

Nguyen and Al-Mubaid Semantic Similarity Measures Source: MSH
Leacock and Chodorow, 1998 Wu and Palmer, 1994 Path Source: MSH Same Dataset created by Pedersen, et al. Data 29 medical terms pairs Similarity determined by: 9 Medical Coders 3 Physicians Spearman’s Rank Correlation Coefficient Nguyen (Wing) and Al-Mubaid evaluate the semantic similarity methods proposed by Leacock and Chodorow, and Wu and Palmer, their own proposed measure and the path measure using the Medical Subject Headings MSH – the actual MSH source not the source integrated in the UMLS. They use the same dataset as Pedersen and evaluate the measure by determining the correlation between the human annotations using Spearman’s Rank Correlation Coefficient

Comparison with Nguyen and Al-Mubaid
Semantic Similarity Measures Nguyen and Al-Mubaid, 2006 Leacock and Chodorow, 1998 Wu and Palmer, 1994 Path Source: MSH from UMLS 2008AB Relations: PAR/CHD Comparison with human annotations Spearman Rank Correlation Coefficient We compare the correlation results obtained by UMLS Similarity using MSH and the PAR/CHD relations in the 2008AB version of the UMLS

Comparison with Nguyen and Al-Mubaid
Measure Physician Coder path Nguyen and Al-Mubaid 0.63 0.85 UMLS-Similarity 0.49 0.58 Leacock and Chodorow 0.67 0.86 Wu and Palmer 0.65 0.79 0.45 0.54 0.55 The results show that UMLS-Similarity obtains a lower correlation than those reported by Nguyen and Al-Mubaid for each of the semantic similarity measures.

Caviedes and Cimino Semantic Similarity Measure Source: MSH
Conceptual Distance – Rada, et al. Source: MSH Relations: PAR/CHD Data 10 medical terms pairs using following CUIs Digestive system disease: C Peptic esophagitis: C Psychotherapy: C Thirst: C Thoracic duct: C Lastly we compare the results of the UMLS-Similarity package with those obtained by Caviedes and Cimino Caviedes and Cimino evaluate the Conceptual distance measure proposed by Rada, et. Al. on 10 medical terms pairs

Comparison with Caviedes and Cimino
Semantic Similarity Measures Conceptual Distance Originally proposed by Rada, et. al., 1989 Source: MSH from UMLS 2008AB Relations: PAR/CHD Comparison between the Conceptual Distance Scores Caviedes and Cimino use the PAR/CHD relations in MSH to determine the Conceptual Distance for each pair of terms. We compare the actual conceptual distance score obtained by Caviedes and Cimino with those obtained by UMLS-Similarity using MSH and the PAR/CHD relations from the UMLS version 2008AB.

Comparison with Caviedes and Cimino
CUI Pairs Caviedes and Cimino UMLS-Similarity C C 3 C C 5 C C 6 C C 7 C C C C 8 9 C C C C 10 C C 11 C C The results show that the UMLS-Similarity obtains the same conceptual distance as Caviedes and Cimino for six out of the ten pairs. The remaining four pairs are only off by one point.

Results The results show that UMLS-Similarity can be used to reproduce the results reported by: Pedersen, et al. Caviedes and Cimino The results show that UMLS-Similarity can be used to reproduce the results reported by Pedersen et al and Caviedes and Cimino We believe that the minor differences in results are due to different versions of the source vocabularies that were used

Results The correlation results obtained by UMLS- Similarity and reported by Nguyen and Al- Mubaid vary Different versions of MSH were used to conduct the experiment Possibly different mappings of the terms to CUIs in MSH were used Information used by Nguyen and Al-Mubaid comes directly from MSH which is located in MRHEIR and as PAR/CHD relations in MRREL It is not possible to generate MRHIER from MRREL because the full path-to-root is a transitive closure of the pairwise PAR/CHD relations which does not hold true for MSH because a MSH concept may have different children depending on its tree position We believe the difference in the results is due to : Different versions of MSH being used to conduct the experiment Possibly different mappings of the terms to CUIs in MSH The path information used by Nguyen (Wing) and Al-Mubaid to determine the semantic similarity comes directly from MSH which in the UMLS would be located in the MRHEIR and the PAR/CHD relations in MRREL But with MSH, the exact path-to-root information for each of the concepts can not be generated in MRREL because a MSH concept may have different children depending on its position in the hierarchy

Conclusions UMLS-Similarity UMLS-Interface
Used to determine the similarity between two concepts given a specified set of sources and relations Contains the following similarity measures Path measure Conceptual Distance proposed Rada, et. al. 1989 Leacock and Chodorow, 1998 Wu and Palmer, 1994 Nguyen and Al-Mubaid, 2006 UMLS-Interface Used to obtain path information about a CUI given a specified set of sources and relations So In conclusion the UMLS-Interface can be used to obain the path information about a CUI given a specified set of sources and relations UMLS-Similarity can be used to determine the semantic similarity between two CUIs in the UMLS Currently, there are five path-based semantic similarity measures included in the package: Conceptual Distance proposed by Rada, et. Al. The measures proposed by Leacock and Chodorow, Wu and Palmer and Nguyen and Al-Mubaid and the path measure

Future Work UMLS-Interface UMLS-Similarity
Improve the efficiency in which the path information is stored UMLS-Similarity Information Content Similarity Measures Resnik, 1995 Jiang and Conrath, 1997 Lin, 1997 Relatedness Measures Patwardhan, 2003 In the future, For UMLS-Interface, we plan to improve the efficiency in which the path information of a CUI is stored And For UMLS-Similarity, we plan to incorporate other semantic similarity and relatedness measures such as the information-content semantic similarity measures proposed by Resnik, Jiang and Conrath and Lin And the semantic relatedness measure proposed by Patwardhan.

set of sources and relations.
Take Home Message #1 UMLS-Interface can used to extract path information about a concept given a specified set of sources and relations. So, this wraps up my presentation about UMLS-Interface and UMLS-Similarity. I have two take home messages that I hope that you remember from this talk. The first is that UMLS-Interface can reliably be used to extract path information about a CUI given a specified set of sources and relations

Take Home Message #2 UMLS-Similarity can be used to compute the semantic similarity between two concepts given a specified set of sources and relations. The second is that UMLS-Similarity can reliably be used to determine the semantic similarity between two CUIs

Availability UMLS-Interface UMLS-Similarity
UMLS-Similarity UMLS-Interface and UMLS-Similarity can both be downloaded from CPAN. I do have a demo set up on my laptop so if you are interested please let me know and I can show you during the break. Thank you - are there any questions

Thank You We would like to thank
Kin Wah Fung, Olivier Bodenreider, Jan Willis Lan Aronson The research was supported in parts by: Fellowships: NLM Research Participation Program GAANN fellowship from US Dept. of Ed. Grants IR01LM A2 from NIH, NLM

Bridget McInnes Ted Pedersen Serguei Pakhomov

Similar presentations

Presentation on theme: "Bridget McInnes Ted Pedersen Serguei Pakhomov"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Bridget McInnes Ted Pedersen Serguei Pakhomov

Similar presentations

Presentation on theme: "Bridget McInnes Ted Pedersen Serguei Pakhomov"— Presentation transcript:

Similar presentations

About project

Feedback