CS 564: Database Management Systems BadgerDB Goal: Build key components of a RDBMS First hand experience building the internals of a simple database system And have some fun doing so! Two parts Buffer manager(Due Date : Feb 17 by 2PM) B+tree All projects are individual assignments 11/10/2018 CS 564: Database Management Systems

Structure of Database Query optimizer and execution Relational operators File and access methods Buffer manager I/O manager Problem it is trying to solve: Disks are slow, especially when it comes to the random access, memory is fast but volatile. But the whole reason we are using disk is to PERSIST things. Imagine a database that promises you to have saved things or committed a transaction but doesn’t. Can we come up with a strategy and mechanism to solve this problem? Note that this nature of problem is not specific to database. Webapplications have similar problems. 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

CS 564: Database Management Systems, Jignesh M. Patel Plan for today Logistics Quick review of pointer and reference BadgerDB: BufferManager Specifications and Code Q&A 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

Pointers and references

Address-of operator (&) foo = &myvar assign the address of variable myvar to foo

Address-of operator (&) foo = &myvar assign the address of variable myvar to foo myvar = 25; foo = &myvar; bar = myvar;

Address-of operator (&) foo = &myvar assign the address of variable myvar to foo myvar = 25; foo = &myvar; bar = myvar; pointer int *

Dereference operator (*) baz = *foo;

Dereference operator (*) baz = *foo;

Dereference operator (*) baz = *foo; baz = foo; baz = *foo;

Dereference operator (*) baz = *foo; baz = foo; // baz equal to foo (1776) baz = *foo; // baz equal to value pointed to by foo (25)

References (&) C++ uses & to denote the address-of operator in an expression C++ assigns an additional meaning to & in declaration to declare a reference variable.

References (&) C++ uses & to denote the address-of operator in an expression C++ assigns an additional meaning to & in declaration to declare a reference variable. int main() { int number = 88; int & refNumber = number; cout << number << endl; // (88) cout << refNumber << endl; // (88) }

References (&) C++ uses & to denote the address-of operator in an expression C++ assigns an additional meaning to & in declaration to declare a reference variable. int main() { int number = 88; int & refNumber = number; cout << number << endl; // (88) cout << refNumber << endl; // (88) refNumber = 99 }

References (&) C++ uses & to denote the address-of operator in an expression C++ assigns an additional meaning to & in declaration to declare a reference variable. int main() { int number = 88; int & refNumber = number; cout << number << endl; // (88) cout << refNumber << endl; // (88) refNumber = 99 cout << number << endl; // ? }

References (&) C++ uses & to denote the address-of operator in an expression C++ assigns an additional meaning to & in declaration to declare a reference variable. int main() { int number = 88; int & refNumber = number; cout << number << endl; // (88) cout << refNumber << endl; // (88) refNumber = 99 cout << number << endl; // (99) }

References (&) Likewise with function signatures accepting references C++ uses & to denote the address-of operator in an expression C++ assigns an additional meaning to & in declaration to declare a reference variable. int main() { int number = 88; int & refNumber = number; cout << number << endl; // (88) cout << refNumber << endl; // (88) refNumber = 99 cout << refNumber << endl; cout << number << endl; // (99) number = 110; cout << number << endl; cout << refNumber << endl; // (110) } Likewise with function signatures accepting references

BadgerDB: BufferManager

Structure of Database Query optimizer and execution Relational operators File and access methods Buffer manager I/O manager 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

CS 564: Database Management Systems BadgerDB: IO Layer Allows upper level layer (Buffer Manager) to Create/destroy files Allocate/delete pages Read/write pages Provided! 11/10/2018 CS 564: Database Management Systems

CS 564: Database Management Systems BadgerDB: IO Layer File Class(File.h) Page allocatePage() create open Page readPage(pageNo) remove void writePage(newPage) isOpen exists Void deletePage(pageNo) filename = Page Class(Page.h) 11/10/2018 CS 564: Database Management Systems

CS 564: Database Management Systems, Jignesh M. Patel Buffer Pool (frames) In-memory buffer holds database pages that are read from disk P1 from File1 <empty> P4 from File2 unit of transfer = “page” File1 P1 P2 P3 P4 P5 P6 P7 File2 P1 P2 P3 P4 P5 P6 P7 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

What does Buffer Manager do? Control which pages are memory resident Request Page N From File M Buffer Manager P1 from File1 <empty> P4 from File2 If not: choose a frame , reads in the pages from disk into the frame If it is in the buffer pool: return! 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

CS 564: Database Management Systems Clock Algorithm Requested page in Buffer Pool: pinCnt++/ Return handle to frame Else read page in from disk find space in the buffer pool! Frame b/k: pinCnt dirty refbit … Read Page (1, T, 0) (2, T, 1) (2, T, 0) Page unpinned (3, F, 0) (0, F, 0) (1, F, 1) (0, F, 1) (0, T, 0) 11/10/2018 CS 564: Database Management Systems

The Clock Replacement Algorithm Advance Clock Pointer No Valid set? Yes refbit set? Yes Clear refbit No Yes Page pinned? No Dirty bit set? Yes Flush page to disk No Call “Set()” on the Frame Use Frame 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

Structure of Buffer Manager BufHashTbl Class Keeps mapping information (file, page number) <-> (buffer pool frame) // insert entry into hash table mapping (file, pageNo) to frameNo void insert(const File* file, const int pageNo, const int frameNo);   // Check if (file,pageNo) is currently in the buffer pool (ie. in // the hash table. If so, assign the corresponding frame number in frameNo. bool lookup(const File* file, const int pageNo, int& frameNo); // remove entry obtained by hashing (file,pageNo) from hash table. void remove(const File* file, const int pageNo); 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

Structure of Buffer Manager BufDesc Class Keeps track of the state of each frame in the buffer pool class BufDesc { private: File* file; // pointer to file object PageId pageNo; // page within file FrameId frameNo; // buffer pool frame number int pinCnt; // number of times this page has been pinned bool dirty; // true if dirty; false otherwise bool valid; // true if page is valid bool refbit; // true if this buffer frame been referenced recently   void Clear(); // initialize buffer frame void Set(File* filePtr, PageId pageNum); //set BufDesc member variable values void Print() //Print values of member variables BufDesc(); //Constructor 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

Structure of Buffer Manager BufMgr Class (buffer.cpp) This is where you write your code class BufMgr { private: FrameId clockHand; // clock hand for clock algorithm BufHashTbl *hashTable; // hash table mapping (File, page) to frame number BufDesc *bufDescTable; // BufDesc objects, one per frame std::uint32_t numBufs; // Number of frames in the buffer pool public: Page* bufPool; 11/10/2018 CS 564: Database Management Systems

CS 564: Database Management Systems, Jignesh M. Patel Assignment Tasks Complete member functions of BufMgr class ~BufMgr(); void advanceClock(); void allocBuf(FrameId& frame); void readPage( File *file, const PageId PageNo, Page*& page); void unPinPage( File *file, const PageID PageNo, const bool dirty) void allocPage( File * file, PageID & PageNo, Page *& Page) void disposePage(File * file, const PageId PageNo) void flushFile(File *file) 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel

CS 564: Database Management Systems, Jignesh M. Patel Main function tests Test1~2 : allocPage, unPinPage, readPage Test3 : reading a page from a invalid file InvalidPageException should be thrown Test4 : unpinning a page whose pinCnt is already zero PageNotPinnedException should be thrown Test5 : allocating too many buffers BufferExceededException should be thrown Test6 : flushing file when it is still being used PagePinnedException should be thrown 11/10/2018 CS 564: Database Management Systems, Jignesh M. Patel