1. Major Issues n Information is mostly online n Information is increasing available in full-text (full-content) n There is an explosion in the amount of information being produced. 1

2. Manual Indexing n Human’s read and index content. n Fairly good, although not consistent (interobserver, or even intraobserver). n Certain fields support costly manual indexing (primary example is Medline). 2

3. Major Issues n For all fields unable to afford manual indexing, and even for biomedical, because there is so much knowledge in the huge amount of literature being produced that we cannot keep track of it, or utilize it. n Research Example: Swanson’s undiscovered literature 3

4. What this means n Need ways to index without requiring paid experts –Automatic indexing, classification, keyword extraction, and even relationship and fact extraction. –Need to take advantage of experts who are reading the materials to comment on it and provide rankings, summarizations, keywords, “factoids”. (like Amazon)Amazon 4

5. 5 Why Automatic Classification? n Classification is time consuming and expensive n Knowledge structuring –To much information n Status of automatic classification –Approaching level of human indexing. (NLM’s Metamap).NLM’s Metamap

6. 6 What is Automatic Classification? n Automatic manipulation of a document’s contents to support logical grouping with other similar documents for organization and/or retrieval activities. Can include the assignment of, or manipulation of, classification notation.

7. 7 Approaches and Methods n Initial approach –Create an inverted file –On-the-fly (natural language processing) n Methods –All words, remove stop words –Word frequencies (Wilson’s objective method of determining aboutness) –More sophisticated IR methods Semantic/linguistical analysis, co-occurrence/similarity measures, etc.

8. 8 Simple automatic indexes n Inverted file: contains all the index terms automatically drawn from the document records according to the indexing technique used. –Position of term -record number -Field number -Number of occurrences -Position in the field (digits 45-57)

9. 9 Pros and Cons of Automatic Indexing n Pros –Consistency –Cost reduction –Time reduction n Cons / limitations –Human intellect –Term relationships –Misleading in retrieval –Good algorithms, but generally domain-specific

10. 10 How to gauge effectiveness? Recall Number of relevant documents retrieved out of all the possible relevant documents in system. [quantity—did you get it all?] Precision Percentage of documents retrieved that were relevant [quality of what you found]

11. 11 Tradeoff between Recall and Precision We can easily recall everything that matches a particular text string or pattern; however, we cannot search through all the matching results (too many) We can do an OK job limiting to most relevant, but as we “tune” result to be more relevant, we leave out more and more matching results.

12. Future Search n Full text searching of content, and of associated annotations on content, and metadata (including reader rankings, tags, etc). Like Connotea, NeoNote, etc.Connotea n Faceted based searching (Endeca, e.g. Home Depot, NCSU library). Home DepotNCSU library n Clustered based searching (Clusty)Clusty 12

13. Study on gene name searching n Looks at full text searching n Tradeoff between precision and recall n (Hemminger 2007). 13

14. 14 Article Discovery Study Schizophrenia + Schizophrenia Gene Schizophrenia GeneArabidopsis Gene Genes Found in Metadata Only 172 8.58%354120.63%27128.83% Genes Found in Full- text Only 167183.38 % 1012558.99%570518.57% Genes Found in Metadata and Full-text 1618.03%349820.38%2230572.60% Totals for Found Genes 20041716430722

15. 15 Article Review Study n Two literature cohorts, –Schizophrenia (Pat Sullivan) –Arabidopsis (Todd Vision) n Each cohort had three readers n Readers are asked to “review the article and judge its relevance to them as someone new to the gene in this biological setting, trying to build an understanding of the state of knowledge in that research area.”

16. 16 Metadata Articles More Valuable n In both cases and for all observers, their mean quality rating values were lower (more useful) for the metadata discovered articles. There were statistically significant differences between the mean quality rating for the metadata discovered articles versus the full-text discovered articles for the both the Arabidopsis and Schizophrenia sets at the p < 0.05 level

17. 17 Precision and Recall SchizophreniaArabidopsis RecallPrecisionRecallPrecision Metadata discovered15.7% (16.6%) 94.7%84.1% (84.1%) 100% Full-text only discovered100%63.7%100%69%

18. 18 Article Features that correlate with Value: Number of Hits n The number of hits or matches of the search term within the returned document is a commonly used feature to rank returned articles. To test the value of this feature, the number of hits was correlated with the mean quality ranking for each article (averaged across all observers). The results clearly show a relationship where articles with many matches of the search term, tend to be much more highly valued.

19. 19 Improving Relevance for Metadata Searching n Repeating the calculations on the schizophrenia and Arabidopsis article review sets, but limited to only matches with high hit counts (Schizophrenia ≥ 20 hits and Arabidopsis ≥ 15 hits) shows that precision for the full text is now the same (100% in Aradidopsis) or slightly better than that of the metadata retrieved articles (95% versus 94.4% in schizophrenia). However, the number of additional cases discovered by full- text searching is now only slightly better, finding 5% more cases in schizophrenia and 28% more in Arabidopsis.

20. 20 Conclusions n This suggests that rather than accepting metadata searching as a surrogate for full- text searching, it may be time to make the transition to direct full text searching as the standard. This could be accomplished by using certain features of the full-text article, such as number of hits of the search string or whether the search string is found in the metadata (i.e. our current metadata search) as filters that allow us to increase the precision of our results. (and put the user in control of the filtering).