1. Vocabulary & languages in searching
© Tefko Saracevic. Rutgres University
Vocabulary & languages in searching
Connection:
indexing
searching
© Tefko Saracevic

2. Basic assertion Indexing and searching: inexorably connected
you cannot search that that was not first indexed in some manner or other
indexing of documents or objects is done in order to be searchable
there are many ways to do indexing
to index one needs an indexing language
there are many indexing languages
even taking every word in a document is an indexing language
Knowing searching is knowing indexing
© Tefko Saracevic

3. General definitions Vocabulary [Encarta Dictionary] Language
“1. words known
LANGUAGE - all the words used by or known to a particular person or group, or contained in a language as a whole”
Language
“1. speech of group
the speech of a country, region, or group of people, including its diction, syntax, and grammar
2. system of communication
a system of communication with its own set of conventions or special words”
© Tefko Saracevic

4. From general to specific
These general definitions are valid for application in indexing & searching to define
index terms
indexing vocabulary
indexing language
search terms
search vocabulary
query (request, search) language
© Tefko Saracevic

5. Specific
Index term
a word or phrase that denotes (describes) a concept & connotes (implies) a class
index term “table” describes a
and implies many kinds of tables:
for which, if desired, we may have more specific index terms
© Tefko Saracevic

6. Specific ... Indexing vocabulary Indexing language
a set of index terms used in a domain or for a set of documents or objects
it could be even a single document or object e.g. a book
Indexing language
an indexing vocabulary together with rules – syntax, grammar – for their application and use
© Tefko Saracevic

7. Specific ... Search terms Search vocabulary Query language
a counterpart to index terms, also denoting a concept and connoting a class for a search
Search vocabulary
a set of search terms in a domain or available in a systems
Query language
a search vocabulary together with rules for their use in searching
© Tefko Saracevic

8. More
“An index language is the language used to describe documents and requests.
The elements of the index language are index terms, which may be derived from the text of the document to be described, or may be arrived at independently.
The vocabulary of an index language may be controlled or uncontrolled.”
(van Rijsbergen, 1979)
© Tefko Saracevic

9. Controlled vocabulary
Predetermined – indicating what terms to be used in indexing
may show definition of and relations between terms
examples: thesaurus, subject heading list, classification
Also indicates terms that may be selected for searching
An indexing AND a searching tool
Human constructed
and costly to construct and use
© Tefko Saracevic

10. Uncontrolled vocabulary
Derived from documents
nowadays automatically
using various ways or algorithms
constant issue: which way is “better”
Used to construct inverted indexes
a concordance, such as of the Bible, indicating place and position of each word mentioned in the text is an inverted index
monks used to do it in 12th century, computers do it today
Inverted indexes are used for free text searching
© Tefko Saracevic

11. Controlled vs. free text searching
Endless source of debate & controversy
But, each has its place for given circumstance & retrieval goal
Each has strengths & weaknesses
can you list or find a list comparing them?
Users mostly use free text searching
Professional searchers use both as warranted
As option:
KNOW THY CONTROLLED VOCABULARY
© Tefko Saracevic

12. Inverted indexes
Useful to know how they function to understand search & retrieval. Steps:
Each document is indexed
every word in a document is taken as index term with exception of stop words
position in text is noted
Indexes for all documents are merged
index terms are arranged alphabetically in the bowel of the system
under each index term are document numbers in which it appears & position in text for that document
© Tefko Saracevic

13. So, when you search for digital AND libraries:
computer takes all documents under digital
and all documents under libraries
compares to “see” which documents have both terms and then
provides you the list of those documents in a default format or you may choose a format
This is also called “coordinate indexing”
coordination is done at time of searching
© Tefko Saracevic

14. Variation: when you search
for digital (WITH) libraries or
“digital libraries” i.e as a phrase
computer goes through the same steps as before but then also
“looks” for documents where digital is positioned right before libraries
remember: computer “knows” position of each term in each document, each sentence
So searching for a phrase is a form of searching of terms connected with AND but in a given sequence
© Tefko Saracevic

15. Example of inverted file
Doc #
Text 1
Slow brown truck arrived 2
Shipment of brownies damaged in a fire 3
Delivery of brownies arrived in a slow truck 4
Shipment of brownies arrived in a truck
For simplicity documents have one sentence. Stop words: “a,” “of,” “in.”
Inverted index
Term
Position in doc number
arrived
(1:4), (3:4), (4:4)
brown
(1:2)
brownies
(2:3), (3:3), (4:3)
damaged
(2:4)
delivery
(3:1)
fire
(2:7)
shipment
(2:1), (4:1)
slow
(1:1), (3:7)
truck
(1:3), (3:8), (4:7)
Search for slow AND truck gets as results documents 1 and 3 since both contain slow and truck
Search for slow (w) truck retrieves only document 3 in which slow is 7th and truck is 8th, they are right next to each other. Doc 1 has both words, but not next to each other thus not retrieved
© Tefko Saracevic

16. Thesaurus
Good old Peter Mark Roget had a most useful idea & did a great job
Following this idea thesaurus became THE major tool for controlled vocabulary in information retrieval (IR)
starting in 1950’s & to this day many IR thesauri have been developed
all have a similar structure & function
but they are difficult & costly to construct
© Tefko Saracevic

17. What is a thesaurus?
“For writers, it is a tool like Roget’s ­ one with words grouped and classified to help select the best word to convey a specific nuance of meaning.
For indexers and searchers, it is an information storage and retrieval tool: a listing of words and phrases authorized for use in an indexing system, together with relationships, variants and synonyms, and aids to navigation through the thesaurus.”
(Milstead, 2000)
© Tefko Saracevic

18. more…
“A thesaurus to an information scientist is a controlled set of the terms used to index information in a database, and therefore also to search for information in that database so the same concepts are represented by the same term.”
(Batty, 1998)
© Tefko Saracevic

19. Basic thesaurus components
For each entry thesaurus has a classification grid:
Descriptor (DE) – an index term that has
Scope note (SN) – context in which used
Broader terms (BT) – higher in a hierarchy
Narrower terms (NT) – lower in a hierarchy
Related terms (RT) – other connected descriptors
Used for (UF) – synonyms that are not descriptors
Note: not all of these may be present for every descriptor
A searcher or indexer can use these as a guide for selection/rejection & for browsing to get ideas
© Tefko Saracevic

20. Examples of thesauri
Thesauri have been constructed for great many domains, from A to Z
here are some lists
international & multilingual thesauri
online thesauri
among them ERIC Thesaurus (we use it for example)
BUT: different thesauri may and do treat the same descriptor (index term) differently
having different, more or fewer narrower, broader, related terms
thus it is dangerous to use them interchangeably
© Tefko Saracevic

21. Standard structure
With variations on the theme, thesauri have similar conceptual structure to guide searcher or indexer:
Descriptor - DE
Broader terms - BT
Narrower terms - NT
Related terms - RT
Used for - UF
Synonyms
Scope note - SN
Note: Every descriptor doesn't have to have all of these
© Tefko Saracevic

22. Same thesaurus but …
Examples of ERIC (Educational Resources Information Center) thesaurus as used differently in different systems:
ERIC own system
ERIC file on DIALOG (begin 1)
ERIC file on OVID (accessible through RUL)
Notice how each uses thesaurus displays & search in its own way, but principles still the same
Oh well…
© Tefko Saracevic

23. ERIC online thesaurus on ERIC
Allows for
searching for words that are included in descriptors by category or all categories
browsing alphabetically
browsing in one of about 40 categories
Search for library in all categories found 76 descriptors that have “library” included
Out of these selected library education
© Tefko Saracevic

24. ERIC online thesaurus on ERIC descriptor library education
© Tefko Saracevic

25. ERIC thesaurus on DIALOG
In a convoluted way ERIC thesaurus (and other ones) can be displayed on DIALOG (and other vendors, such as OVID)
How?
begin in file 1 – ERIC
then expand a desired term – here we used term library
you will see under R that certain terms have related terms – meaning that these are thesaurus entries
then expand on one of those to see related terms
then you can browse & choose which ones to use in search
And here are Print Screens of the process
© Tefko Saracevic

26. going …
Expand
library
© Tefko Saracevic

27. going … RT indicates related terms 45237 items have library
This one has 14 related terms
going …
© Tefko Saracevic

28. You can expand on expand
going …
We now chose descriptor LIBRARY ADMINISTRATION and expand on that one
Neat trick:
You can expand on expand
& get related terms
© Tefko Saracevic

29. going … These are now R terms of various type
14 related terms for this one are listed
Can expand on this one to see other RT
You can also select any of these to search
© Tefko Saracevic

30. going … We have now selected r10 – library expenditures
© Tefko Saracevic

31. going … And this is what we got
Now we can view some items in a chosen format
or we can further modify this search - add refine, …
© Tefko Saracevic

32. gone This is one of the items we got Descriptors with * are major
Additional index terms
Descriptors used for this item
Descriptors with * are major
© Tefko Saracevic

33. ERIC thesaurus on OVID (accessed through RUL)
For library ask to map as thesaurus term
© Tefko Saracevic

34. There are more down there but we choose this one to expand
going …
There are more down there but we choose this one to expand
© Tefko Saracevic

35. going … Entries for descriptor Electronic Libraries
Continue to search for AND
© Tefko Saracevic

36. Retrieved & ready to display
going …
Retrieved & ready to display
© Tefko Saracevic

37. Choose format you want for this item
gone
Choose format you want for this item
© Tefko Saracevic

38. Relevance feedback
Method for using information in items judged relevant to further refine or change the search
e.g. in relevant items we can browse titles, descriptors, identifiers, abstracts … to get leads for further search terms & tactics
in some advanced systems this may be done automatically
© Tefko Saracevic

39. Query expansion
Method for adding, modifying, changing search terms in query
to broaden, narrow, focus, change … terms
Many sources can be used
relevance feedback, thesauri, dictionaries, textbooks, documents, catalogs, & people: users, colleagues, your own mind & experience
Some systems suggest terms for query expansion
© Tefko Saracevic

40. Conclusion At the base of all searching are
terms
vocabularies
languages
but a variety exists
In reality in searching there is no completely controlled or uncontrolled vocabulary
matter of degree
& most importantly, matter of mastery
© Tefko Saracevic

41. symbolically; controlled & free vocabulary
© Tefko Saracevic

42. thank you!
© Tefko Saracevic