1. Chap 7. Indexing

2. Chapter Objectives(1)
Introduce concepts of indexing that have broad applications in the design of file systems
Introduce the use of a simple linear index to provide rapid access to records in an entry-sequenced, variable-length record file
Investigate the implementation of the use of indexes for file maintenance
Introduce the template features of C++ for object I/O
Describe the object-oriented approach to indexed sequential files

3. Chapter Objectives(2)
Describe the use of indexes to provide access to records by more than one key
Introduce the idea of an inverted list, illustrating Boolean operations on lists
Discuss of when to bind an index key to an address in the data file
Introduce and investigate the implications of self-indexing files

4. Contents(1) 7.1 What is an Index?
7.2 A Simple Index for Entry-Sequenced Files
7.3 Using Template Classes in C++ for Object I/O
7.4 Object-Oriented Support for Indexed, Entry Sequenced Files of Data Objects
7.5 Indexes That Are Too Large to Hold in Memory

5. Contents(2) 7.6 Indexing to Provide Access by Multiple Keys
7.7 Retrieval Using Combinations of Secondary Keys
7.8 Improving the Secondary Index Structure: Inverted Lists
7.9 Selective Indexes
7.10 Binding

6. Overview: Index(1)
Index: a data structure which associates given key values with corresponding record numbers
It is usually physically separate from the file (unlike for indexed sequential files tight binding).
Linear indexes (like indexes found at the back of books)
Index records are ordered by key value as in an ordered relative file
Best algorithm for finding a record with a specific key value is binary search
Addition requires reorganization

7. Overview: Index(2) Index File Data File k1 k2 k4 k5 k7 k9 AAA ZZZ CCC
XXX
EEE
FFF
Index File
Data File

8. Overview: Index(3)
Tree Indexes (like those of indexed sequential files)
Hierarchical in that each level
Beginning with the root level, points to the next record
Leaves POINTs only the data file
Indexed Sequential File
Binary Tree Index
AVL Tree Index
B+ tree Index

9. Roles of Index? Index: keys and reference fields Fast Random Accesses
Uniform Access Speed
Allow users to impose order on a file without actually rearranging the file
Provide multiple access paths to a file
Give user keyed access to variable-length record files

10. A Simple Index(1) Datafile Search a file with key (popular need)
entry-sequenced, variable-length record
primary key : unique for each entry in a file
Search a file with key (popular need)
cannot use binary search in a variable-length record file(can’t know where the middle record)
construct an index object for the file
index object : key field + byte-offset field

11. A Simple Index (2) Datafile Indexfile Key ANG3795 167 COL31809 353DG FF
LON
MER
RCA
WAR DG
LON|2312|Romeo and Juliet|Prokofiev . . .
RCA|2626|Quarter in C Sharp Minor . . .
WAR|23699|Touchstone|Corea . . .
ANG|3795|Sympony No. 9|Beethoven . . .
COL|38358|Nebeaska|Springsteen . . .
DG|18807|Symphony No. 9|Beethoven . . .
MER|75016|Coq d'or Suite|Rimsky . . .
COL|31809|Symphony No. 9|Dvorak . . .
DG|139201|Violin Concerto|Beethoven . . .
FF|245|Good News|Sweet Honey In The . . . 32 77
132
167
211
256
300
353
396
442
Datafile
Actual data record
Address of
record
Reference
field
Key
Indexfile

12. A Simple Index (3) Key Reference field
Index file: fixed-size record, sorted
Datafile: not sorted because it is entry sequenced
Record addition is quick (faster than a sorted file)
Can keep the index in memory
find record quickly with index file than with a sorted one
Class TextIndex encapsulates the index data and index operations

13. Let’s See Figure 7.4 Class TextIndex{ public:
TextIndex(int maxKeys = 100, int unique = 1);
int Insert(const char*ckey, int recAddr); //add to index
int Remove(const char* key); //remove key from index
int Search(const char* key) const;
//search for key, return recAddr
void Print (ostream &) const;
protected:
int MaxKeys; // maximum num of entries
int NumKeys;// actual num of entries
char **Keys; // array of key values
int* RecAddrs; // array of record references
int Find (const chat* key) const;
int Init (int maxKeys, int unique);
int Unique;// if true --> each key must be unique }

14. Constructor & Destructor
TextIndex:: TextIndex (int maxKeys, int unique)
: NumKeys (0), Keys(0), RecAddrs(0)
{Init (maxKeys, unique);}
int TextIndex :: Init (int maxKeys, int unique) {
Unique = unique != 0;
if (maxKeys <= 0)
MaxKeys = 0;
return 0; }
MaxKeys = maxKeys;
Keys = new char *[maxKeys];
RecAddrs = new int [maxKeys];
return 1;
TextIndex :: ~TextIndex ()
{delete Keys; delete RecAddrs;}

15. Insert
int TextIndex :: Insert (const char * key, int recAddr) { int i; int index = Find (key); if (Unique && index >= 0) return 0; // key already in if (NumKeys == MaxKeys) return 0; //no room for another key for (i = NumKeys-1; i >= 0; i--) if (strcmp(key, Keys[i])>0) break; // insert into location i+1 Keys[i+1] = Keys[i]; RecAddrs[i+1] = RecAddrs[i]; } Keys[i+1] = strdup(key); RecAddrs[i+1] = recAddr; NumKeys ++; return 1;

16. Remove
int TextIndex :: Remove (const char * key) { int index = Find (key); if (index < 0) return 0; // key not in index for (int i = index; i < NumKeys; i++) Keys[i] = Keys[i+1]; RecAddrs[i] = RecAddrs[i+1]; } NumKeys --; return 1;

17. Search
int TextIndex :: Search (const char * key) const { int index = Find (key); if (index < 0) return index; return RecAddrs[index]; } int TextIndex :: Find (const char * key) const for (int i = 0; i < NumKeys; i++) if (strcmp(Keys[i], key)==0) return i;// key found else if (strcmp(Keys[i], key)>0) return -1;// not found return -1;// not found

18. Index Implementation Page 706~709 G.1 Recording.h G.2 Recording.cpp
G.3 Makere.cpp
Page 710~712
G.4 Textind.h
G.5 Textind.cpp

19. RetrieveRecording with the Index
RetrieveRecording(KEY...) procedure : retrieve a single record by key from datafile. And puts together the index search, file read, and buffer unpack operations into single function
int RetriveRecording (Recording & recording, char * key,
TextIndex & RecordingIndex, BufferFile & RecordingFile)
// read and unpack the recording, return TRUE if succeeds
{ int result;
result = RecordingFile . Read (RecordingIndex.Search(key));
if (result == -1) return FALSE;
result = recording.Unpack (RecordingFile.GetBuffer());
return result; }

20. Template Class for I/O Object(1)
Template Class RecordFile
we want to make the following code possible
Person p; RecordFile pFile; pFile.Read(p);
Recording r; RecordFile rFile; rFile.Read(r);
difficult to support files for different record types without having to modify the class
Template class which is derived from BufferFile
the actual declarations and calls
RecordFile <Person> pFile; pFile.Read(p);
RecordFile <Recording> rFile; rFile.Read(p);

21. Template Class for I/O Object(2)
Template Class RecordFile
template <class RecType>
class RecordFile : public BufferFile{
public:
int Read(RecType& record, int recaddr = -1);
int Write(const RecType& record, int recaddr = -1);
int Append(const RecType& record);
RecordFile(IOBuffer& buffer) : BufferFile(buffer) {} };
//The template parameter RecType must have the following methods
//int Pack(IOBuffer &); pack record into buffer
//int Unpack(IOBuffer &); unpack record from buffer

22. Template Class for I/O Object(3)
Adding I/O to an existing class RecordFile
add methods Pack and Unpack to class Recording
create a buffer object to use in the I/O
DelimFieldBuffer Buffer;
declare an object of type RecordFile<Recording>
RecordFile<Recording> rFile (Buffer);
Declaration and Calls
Recording r1, r2;
rFile.Open(“myfile”);
rFile.Read(r1);
rFile.Write(r2);
Directly open a file and read and
write objects of class Recording

23. Object-Oriented Approach to I/O
Class IndexedFile
add indexed access to the sequential access provided by class RecordFile
extends RecordFile with Update, Append and Read method
Update & Append : maintain a primary key index of data file
Read : supports access to object by key
TextIndex, RecordFile ==> IndexedFile
Issues of IndexedFile
how to make a persistent index of a file
how to guarantee that the index is an accurate reflection of the contents of the data file

24. Basic Operations of IndexedFile(1)
Create the original empty index and data files
Load the index file into memory
Rewrite the index file from memory
Add records to the data file and index
Delete records from the data file
Update records in the data file
Update the index to reflect changes in the data file
Retrieve records

25. Basic Operations of TextIndexedFile (1)
Creating the files
initially empty files (index file and data file)  created as empty files with header records
implementation ( makeind.cpp in Appendix G )  Create method in class BufferFile
Loading the index into memory
loading/storing objects are supported in the IOBuffer classes
need to choose a particular buffer class to use for an index file ( tindbuff.cpp in Appendix G )
define class TextIndexBuffer as a derived class of FixedFieldBuffer to support reading and writing of index objects

26. Basic Operations of TextIndexedFile(2)
Rewriting the index file from memory
part of the Close operation on an IndexedFile
write back index object to the index file
should protect the index when failure
write changes when out-of-date(use status flag)
Implementation
Rewind and Write operations of class BufferFile
Record Addition
Add a new record to data file
using
RecordFile<Recording>::Write
Add an entry to the index
Requires rearrangement
if in memory, no file access
using TextIndex.Insert +

27. Basic Operations of TextIndexedFile(3)
Record Deletion
data file: the records need not be moved
index: delete entry really or just mark it
using TextIndex::Delete
Record Updating (2 categories)
the update changes the value of the key field
delete/add approach
reorder both the index and the data file
the update does not affect the key field
no rearrangement of the index file
may need to reconstruct the data file

28. Class TextIndexedFile(1)
Members
methods
Create, Open, Close, Read (sequential & indexed), Append, and Update operations
protected members
ensure the correlation between the index in memory (Index), the index file (IndexFile), and the data file (DataFile)
char* key()
the template parameter RecType must have the key method
used to extract the key value from the record

29. Class TextIndexedFile(2)
Template <class RecType>
class TextIndexedFile
{ public:
int Read(RecType& record); // read next record
int Read(char* key, RecType& record) // read by key
int Append(const RecType& record);
int Update(char* oldKey, const RecType& record);
int Create(char* name, int mode=ios::in|los::out);
int Open(char* name, int mode=ios::in|los::out);
int Close();
TextIndexedFile(IOBuffer & buffer, int keySize, int maxKeys=100);
~TextIndexedFile(); // close and delete
protected:
TextIndex Index; BufferFile IndexFile;
TextIndexBuffer IndexBuffer;
RecordFile<RecType> DataFile;
char * FileName; // base file name for file
int SetFileName(char* fName, char*& dFileName, char*&IdxFName); };

30. TextIndexBuffer class TextIndexBuffer: public FixedFieldBuffer
TextIndexBuffer(int keySize, int maxKeys = 100,
int extraFields = 0, int extraSize=0);
// extraSize is included to allow derived classes to extend
// the buffer with extra fields.
// Required because the buffer size is exact.
int Pack (const TextIndex &);
int Unpack (TextIndex &);
void Print (ostream &) const;
protected:
int MaxKeys;
int KeySize;
char * Dummy; // space for dummy in pack and unpack };

31. TextIndexBuffer - Constructor
TextIndexBuffer::TextIndexBuffer (int keySize, int maxKeys, int extraFields, int extraSpace) : FixedFieldBuffer (1+2*maxKeys+extraFields, /*max fields*/ /* max bytes*/ sizeof(int)+maxKeys*keySize+maxKeys*sizeof(int) + extraSpace) // buffer fields consist of // numKeys, actual number of keys // Keys [maxKeys] key fields size = maxKeys * keySize // RecAddrs [maxKeys] record address fields size = maxKeys*sizeof(int) { MaxKeys = maxKeys; KeySize = keySize; AddField (sizeof(int)); for (int i = 0; i < maxKeys; i++) AddField (KeySize); } Dummy = new char[keySize+1];

32. TextIndexBuffer - Pack
int TextIndexBuffer::Pack(const TextIndex & index) { int result; Clear (); result = FixedFieldBuffer::Pack (&index.NumKeys); for (int i = 0; i < index.NumKeys; i++) {// note only pack the actual keys and recaddrs result = result && FixedFieldBuffer::Pack (index.Keys[i]); result = result && FixedFieldBuffer::Pack (&index.RecAddrs[i]); } for (int j = 0; j<index.MaxKeys-index.NumKeys; j++) {// pack dummy values for other fields result = result && FixedFieldBuffer::Pack (Dummy); return result;

33. TextIndexBuffer - UnPack
int TextIndexBuffer::Unpack (TextIndex & index) { int result; result = FixedFieldBuffer::Unpack (&index.NumKeys); for (int i = 0; i < index.NumKeys; i++) {// note only pack the actual keys and recaddrs index.Keys[i] = new char[KeySize]; // just to be safe result = result && FixedFieldBuffer::Unpack (index.Keys[i]); result = result && FixedFieldBuffer::Unpack (&index.RecAddrs[i]); } for (int j = 0; j<index.MaxKeys-index.NumKeys; j++) {// pack dummy values for other fields result = result && FixedFieldBuffer::Unpack (Dummy); return result;

34. Class TextIndexedFile(2)
Template <class RecType>
class TextIndexedFile
{ public:
int Read(RecType& record); // read next record
int Read(char* key, RecType& record) // read by key
int Append(const RecType& record);
int Update(char* oldKey, const RecType& record);
int Create(char* name, int mode=ios::in|los::out);
int Open(char* name, int mode=ios::in|los::out);
int Close();
TextIndexedFile(IOBuffer & buffer, int keySize, int maxKeys=100);
~TextIndexedFile(); // close and delete
protected:
TextIndex Index; BufferFile IndexFile;
TextIndexBuffer IndexBuffer;
RecordFile<RecType> DataFile;
char * FileName; // base file name for file
int SetFileName(char* fName, char*& dFileName, char*&IdxFName); };

35. TextIndexFile - Constructor
template <class RecType> TextIndexedFile<RecType>::TextIndexedFile (IOBuffer & buffer, int keySize, int maxKeys) :DataFile(buffer), Index (maxKeys), IndexFile(IndexBuffer), IndexBuffer(keySize, maxKeys) { FileName = 0; }

36. TextIndexFile - Create
template <class RecType> int TextIndexedFile<RecType>::Create (char * fileName, int mode) // use fileName.dat and fileName.ind { int result; char * dataFileName, * indexFileName; result = SetFileName (fileName, dataFileName, indexFileName); if (result == -1) return 0; result = DataFile.Create (dataFileName, mode); if (!result) { FileName = 0; // remove connection return 0; } result = IndexFile.Create (indexFileName, ios::out|ios::in); { DataFile . Close(); // close the data file FileName = 0; // remove connection return 1;

37. TextIndexFile - Open
template <class RecType> int TextIndexedFile<RecType>::Open (char * fileName, int mode) // open data and index file and read index file { int result; char * dataFileName, * indexFileName; result = SetFileName (fileName, dataFileName, indexFileName); if (!result) return 0; result = DataFile.Open (dataFileName, mode); // open files if (!result){ …; return 0;} result = IndexFile.Open (indexFileName, ios::out); // read index into memory result = IndexFile . Read (); if (result != -1){ result = IndexBuffer . Unpack (Index); if (result != -1) return 1;} // read or unpack failed! DataFile.Close(); IndexFile.Close(); FileName = 0; return 0; }

38. TextIndexFile - Read
template <class RecType> int TextIndexedFile<RecType>::Read (RecType & record) { return result = DataFile . Read (record, -1); } int TextIndexedFile<RecType>::Read (char * key, RecType & record) int ref = Index.Search(key); if (ref < 0) return -1; int result = DataFile . Read (record, ref); return result;

39. TextIndexFile - Append
template <class RecType> int TextIndexedFile<RecType>::Append (const RecType & record) { char * key = record.Key(); int ref = Index.Search(key); if (ref != -1) return -1; // key already in file ref = DataFile . Append(record); int result = Index . Insert (key, ref); return ref; }

40. TextIndexFile - Close
template <class RecType> int TextIndexedFile<RecType>::Close () { int result; if (!FileName) return 0; // already closed! DataFile . Close(); IndexFile . Rewind(); IndexBuffer.Pack (Index); result = IndexFile . Write (); cout <<"result of index write: "<<result<<endl; IndexFile . Close (); FileName = 0; return 1; }

41. TextIndexFile - Destructor
template <class RecType> TextIndexedFile<RecType>::~TextIndexedFile () { Close(); }

42. Enhancements to TextIndexedFile(1)
Support other types of keys
Restriction: the key type is restricted to string (char *)
Relaxation: support a template class SimpleIndex with parameter for key type
Support data object class hierarchies
Restriction: every object must be of the same type in RecordFile
Relaxation: the type hierarchy supports virtual pack methods

43. Enhancements to TextIndexedFile(2)
Support multirecord index files
Restriction: the entire index fit in a single record
Relaxation: add protected method Insert, Delete, and Search to manipulate the arrays of index objects
Active optimization of operations
Obvious: the most obvious optimization is to use binary search in the Find method
Active: add a flag to the index object to avoid writing the index record back to the index file when it has not been changed

44. Where are we going? Plain Stream File
Persistency ==> Buffer support ==> BufferFile
<incremental approach> Deriving BufferFile using
various other classes
Random Access ==> Index support => IndexedFile
<incremental approach> : Deriving TextIndexedFile using RecordFile and TextIndex

45. Too Large Index(1) On secondary storage (large linear index)
Disadvantages
binary searching of the index requires several seeks(slower than a sorted file)
index rearrangement requires shifting or sorting records on second storage
Alternatives (to be considered later)
hashed organization
tree-structured index (e.g. B-tree)

46. Too Large Index (2)
Advantages over the use of a data file sorted by key even if the index is on the secondary storage
can use a binary search
sorting and maintaining the index is less expensive than doing the data file
can rearrange the keys without moving the data records if there are pinned records

47. Index by Multiple Keys(1)
DB-Schema = ( ID-No, Title, Composer, Artist, Label)
Find the record with ID-NO “COL38358” (primary key - ID-No)
Find all the recordings of “Beethoven” (2ndary key - composer)
Find all the recordings titled “Violin Concerto” (2ndary key - title)

48. Index by Multiple Keys(2)
Most people don’t want to search only by primary key
Secondary Key
can be duplicated
Figure -->
Secondary Key Index
secondary key --> consult one additional index (primary key index)
BEETHOVEN
DG18807

49. Secondary Index:Basic Operations(1)
Record Addition
similar to the case of adding to primary index
secondary index is stored in canonical form
fixed length (so it can be truncated)
original name can be obtained from the data file
can contain duplicate keys
local ordering in the same key group

50. Secondary Index:Basic Operations (2)
Record Deletion (2 cases)
Secondary index references directly record
delete both primary index and secondary index
rearrange both indexes
Secondary index references primary key
delete only primary index
leave intact the reference to the deleted record
advantage : fast
disadvantage : deleted records take up space

51. Secondary Index: Basic Operations (3)
Record Updating
primary key index serves as a kind of protective buffer
Secondary index references directly record
update all files containing record’s location
Secondary index references primary key (1)
affect secondary index only when either primary or secondary key is changed
Continued.

52. Secondary Index: Basic Operations (4)
Secondary index references primary key(2)
when changes the secondary key
rearrange the secondary key index
when changes the primary key
update all reference field
may require reordering the secondary index
when confined to other fields
do not affect the secondary key index

53. Retrieval of Records Types primary key access secondary key access
combination of above
Combination of keys
using secondary key index, it is easy
boolean operation (AND, OR)

54. Inverted Lists(1) Inverted List
a secondary key leads to a set of one or more primary keys
Disadvantages of 2nd-ary index structure
rearrange when adding
repeated entry when duplicating
Solution A: by an array of references
Solution B: by linking the list of references

55. Secondary key Set of primary key references Revised composer index
Array of References
* no need to rearrange
* limited reference array
* internal fragmentation
BEETHOVEN ANG DG DG RCA2626
COREA WAR23699
DVORAK COL31809
PROKOFIEV LON2312
RIMSKY-KORSAKOV MER75016
SPRINGSTEEN COL38358
SWEET HONEY IN THE R FF245
Secondary key Set of primary key references
Revised composer index

56. Inverted Lists (2) Guidelines for better solution
no reorganization when adding
no limitation for duplicate key
no internal fragmentation
Solution B: by Linking the list of references
A list of primary key references
secondary key field, relative record number of the first corresponding primary key reference
PROKOFIEV
ANG36193
LON2312

57. Linking List of References (1)
BEETHOVEN
COREA
PROKOFIEV
RIMSKY-KORSAKOV
SPINGSTEEN
SWEET HONEY IN THE R
DVORAK 3 2 7 10 6 4 9
LON2312
RCA2626
ANG23699
COL38358
DG18807
MER75016
COL31809
DG139201
ANG36193
WAR23699 -1 8 1 5
FF245
Secondary Index file Label ID List file
Improved revision of the composer index

58. Linking List of References (2)
The primary key references in a separate, entry-sequenced file
Advantages
rearranges only when secondary key changes
rearrangement is quick
less penalty associated with keeping the secondary index file on secondary storage (less need for sorting)
Label ID List file not need to be sorted
reusing the space of deleted record is easy

59. Linking List of References (3)
Disadvantage
same secondary key references may not be physically grouped
lack of locality
could involve a large amount of seeking
solution: reside in memory
same Label ID list can hold the lists of a number of secondary index files
if too large in memory, can load only a part of it

60. Selective Indexes Selective Index: Index on a subset of records
Selective index contains only some part of entire index
provide a selective view
useful when contents of a file fall into several categories
e.g. 20 < Age < 30 and $1000 < Salary

61. Index Binding(1) File construction time binding
When to bind the key indexes to the physical address of its associated record?
File construction time binding
(Tight, in-the-data binding)
tight binding & faster access
the case of primary key
when secondary key is bound to that time
simpler and faster retrieval
reorganization of the data file results in modifications of all bound index files

62. Index Binding (2)
Postpone binding until a record is actually retrieved (Retrieval-time binding)
minimal reorganization & safe approach
mostly for secondary key
Tight, in-the-data binding is good when
static, little or no changes
rapid performance during retrieval
mass-produced, read-only optical disk

63. Let’s Review (1) 7.1 What is an Index?
7.2 A Simple Index for Entry-Sequenced Files
7.3 Using Template Classes in C++ for Object I/O
7.4 Object-Oriented Support for Indexed, Entry Sequenced Files of Data Objects
7.5 Indexes That Are Too Large to Hold in Memory

64. Let’s Review(2) 7.6 Indexing to Provide Access by Multiple Keys
7.7 Retrieval Using Combinations of Secondary Keys
7.8 Improving the Secondary Index Structure: Inverted Lists
7.9 Selective Indexes
7.10 Binding