Annual reports and feedback from UMLS licensees Kin Wah Fung MD, MSc, MA The UMLS Team National Library of Medicine Workshop on the Future of the UMLS Semantic Network

2 Background u One of the requirements of the UMLS license is to submit an annual report of UMLS usage within 30 days of the end of each year u Up to last year, a free-form (with suggested section headings) was the main way to collect the annual reports. Disadvantages: l low response rate (usually less than 100 received each year) l difficult to analyze u From 2005, online form to collect annual reports. Advantages: l more structured questions l easier to analyze l targetted reminders possible => higher response rate

3 Format of online annual report u Logon page – identity authenticated with UMLS license number u Main report page: l user info: affiliation, activity type, computing environment l usage of UMLS l feedback on user support and documentation u Optional sections for feedback on: l Metathesaurus l Semantic Network l Lexical tools l UMLS Knowledge Source Server

4 Results (up to March 31, 2005) Total responses 1,005 Valid responses 988 Have used the UMLS 495 Have used the Semantic Network 210 Provided feedback on the Semantic Network 76

5 Results (up to March 31, 2005) Total responses 1,005 Valid responses 988 Have used the UMLS 495 Have used the Semantic Network 210 Provided feedback on the Semantic Network 76

6 Affiliation Academic 54 (71%) Commercial 12 (16%) Self-employed 4 (5%) Government – non NLM 3 (4%) Government – NLM 2 (3%) Not-for-profit 1 (1%)

7 Main activity of organization/company Research 27 (36%) Education 21 (28%) Software development 11 (14%) Health care provider 10 (13%) Information service 2 (3%) Health insurance 1 (1%) Other 3 (4%) Missing 1 (1%)

8 How long have you been using the UMLS? u Range 2 months – 15 years u Average: 4.3 years u Median 2.3 years u Significantly longer than the rest of respondents - average 2.6 years, median 1 year (Independent-samples T test, 2- tailed p = 0.003)

9 How are you using the UMLS? Terminology research 49 Map between terminologies 39 Indexing and information retrieval 36 Support natural language processing 34 Discover concepts in biomedical texts 32 Acquire biomedical knowledge 32 Design, construct and maintain your terminology 29 Support terminology service 15 Access specific terminologies 15 Publish and distribute your terminology 6 Other2

10 What kind of information do you use the UMLS to process? Clinical information Non-clinical information Findings & diagnosis 46 Genomics & proteomics 23 Procedures31 Bibliographic information 21 Laboratory reports 27 Consumer health information 10 Drugs26Billing3 Clinical guidelines 14Other3 Clinical outcome 10 Nursing information 8 Other8 Total17060

11 Usage of the Semantic Network u How have you used the Semantic Network? l in a research setting – 69 l in a production environment – 7 u Ways in which Semantic Network is used: l in natural language processing n word sense disambiguation n tagging of natural language expressions n natural language generation l ontological uses n finding concept distance n reference for building biomedical ontologies l other uses n suggesting relationships between UMLS concepts n restricting search of terms in the Metathesaurus n research on structure of Semantic Network

12 Components of Semantic Network used u Do you use the Semantic Network on its own or in conjunction with the Metathesaurus concepts? l on its own - 15 l in conjunction - 58 Semantic types 64 (84%) Semantic groups 48 (63%) Semantic relationships (e.g. location_of, causes) 43 (57%) Semantic relations (e.g. Tissue|location_of|Neoplastic Process) 39 (51%)

13 Have you extended or modified the Semantic Network? Semantic Type Semantic Relationship Semantic Relation Area of application 1++ creating terminology 2+ supporting terminology server 3++ building knowledge bases 4++ natural language generation 5+++ creating terminology 6- natural language processing of clinical documents 7- natural language processing of clinical and biomedical documents 8-+/- concept distance estimation 9+/-+/-+/- clinical information systems

14 What improvements would you suggest? u more semantic types (4) u better description of semantic types and how they are assigned (2) u multiple hierarchies u more semantic relations u more uniform granularity u extension of relations to concept level u evolve towards a formal ontology

15 One possible way to prioritize changes?

16 One possible way to prioritize changes? Higher demand Lower demand

17 One possible way to prioritize changes? Higher demand Lower demand Less work involved More work involved

18 One possible way to prioritize changes? Higher demand Lower demand Less work involved - better description of semantic types and how they are assigned (2) - multiple hierarchies - more semantic relations More work involved - more semantic types (4) - more uniform granularity - extension of relations to concept level - evolve towards a formal ontology

19 One possible way to prioritize changes? Higher demand Lower demand Less work involved - better description of semantic types and how they are assigned (2) - multiple hierarchies - more semantic relations More work involved - more semantic types (4) - more uniform granularity - extension of relations to concept level - evolve towards a formal ontology

20 Thank you!