6/27/20151 PSU’s CS 587 13. External Sorting  Motivation  2-way External Sort: Memory, passes,cost  General External Sort: Memory, passes, cost  Optimizations.

Slides:



Advertisements
Similar presentations
External sorting R & G – Chapter 13 Brian Cooper Yahoo! Research.
Advertisements

External Sorting The slides for this text are organized into chapters. This lecture covers Chapter 11. Chapter 1: Introduction to Database Systems Chapter.
Equality Join R X R.A=S.B S : : Relation R M PagesN Pages Relation S Pr records per page Ps records per page.
CS 245Notes 71 CS 245: Database System Principles Notes 7: Query Optimization Hector Garcia-Molina.
CS CS4432: Database Systems II Operator Algorithms Chapter 15.
Database Management Systems 3ed, R. Ramakrishnan and Johannes Gehrke1 Evaluation of Relational Operations: Other Techniques Chapter 14, Part B.
Database Management Systems, R. Ramakrishnan and Johannes Gehrke1 Evaluation of Relational Operations: Other Techniques Chapter 12, Part B.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.
External Sorting “There it was, hidden in alphabetical order.” Rita Holt R&G Chapter 13.
External Sorting CS634 Lecture 10, Mar 5, 2014 Slides based on “Database Management Systems” 3 rd ed, Ramakrishnan and Gehrke.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 Query Evaluation Chapter 11 External Sorting.
FALL 2004CENG 351 Data Management and File Structures1 External Sorting Reference: Chapter 8.
External Sorting R & G Chapter 13 One of the advantages of being
FALL 2006CENG 351 Data Management and File Structures1 External Sorting.
External Sorting R & G Chapter 11 One of the advantages of being disorderly is that one is constantly making exciting discoveries. A. A. Milne.
CS 4432lecture #31 CS4432: Database Systems II Lecture #3 Professor Elke A. Rundensteiner.
Query Processing 1: Joins and Sorting R&G, Chapters 12, 13, 14 Lecture 8 One of the advantages of being disorderly is that one is constantly making exciting.
Query Optimization 3 Cost Estimation R&G, Chapters 12, 13, 14 Lecture 15.
1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.
1 Evaluation of Relational Operations: Other Techniques Chapter 12, Part B.
External Sorting 198:541. Why Sort?  A classic problem in computer science!  Data requested in sorted order e.g., find students in increasing gpa order.
1 External Sorting for Query Processing Yanlei Diao UMass Amherst Feb 27, 2007 Slides Courtesy of R. Ramakrishnan and J. Gehrke.
External Sorting Chapter 13.. Why Sort? A classic problem in computer science! Data requested in sorted order  e.g., find students in increasing gpa.
External Sorting Problem: Sorting data sets too large to fit into main memory. –Assume data are stored on disk drive. To sort, portions of the data must.
Query Processing 2: Sorting & Joins
BTrees & Sorting 11/3. Announcements I hope you had a great Halloween. Regrade requests were due a few minutes ago…
Sorting.
Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 13.
Database Management Systems, R. Ramakrishnan and J. Gehrke 1 External Sorting Chapter 13.
Memory Management during Run Generation in External Sorting – Larson & Graefe.
CPSC 404, Laks V.S. Lakshmanan1 External Sorting Chapter 13 (Sec ): Ramakrishnan & Gehrke and Chapter 11 (Sec ): G-M et al. (R2) OR.
CPSC 404, Laks V.S. Lakshmanan1 External Sorting Chapter 13: Ramakrishnan & Gherke and Chapter 2.3: Garcia-Molina et al.
1 External Sorting. 2 Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing gpa order.
1 Database Systems ( 資料庫系統 ) December 7, 2011 Lecture #11.
Chapter 12 Query Processing (1) Yonsei University 2 nd Semester, 2013 Sanghyun Park.
DMBS Internals I. What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the database concurrently.
ZA classic problem in computer science! zData requested in sorted order ye.g., find students in increasing cap order zSorting is used in many applications.
B+ Trees: An IO-Aware Index Structure Lecture 13.
Query Processing CS 405G Introduction to Database Systems.
Introduction to Database Systems1 External Sorting Query Processing: Topic 0.
Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke1 External Sorting Chapters 13: 13.1—13.5.
CPSC Why do we need Sorting? 2.Complexities of few sorting algorithms ? 3.2-Way Sort 1.2-way external merge sort 2.Cost associated with external.
DMBS Internals I February 24 th, What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the.
DMBS Internals I. What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the database concurrently.
Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 13.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
External Sorting. Why Sort? A classic problem in computer science! Data requested in sorted order –e.g., find students in increasing gpa order Sorting.
External Sorting Jianlin Feng School of Software SUN YAT-SEN UNIVERSITY courtesy of Joe Hellerstein for some slides.
CS 405G: Introduction to Database Systems Instructor: Jinze Liu Fall 2007.
What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the database concurrently and safely. Provide.
Database Systems (資料庫系統)
External Sorting Chapter 13
Database Management Systems (CS 564)
Lecture#12: External Sorting (R&G, Ch13)
External Sorting The slides for this text are organized into chapters. This lecture covers Chapter 11. Chapter 1: Introduction to Database Systems Chapter.
External Sorting Chapter 13
Selected Topics: External Sorting, Join Algorithms, …
CS222P: Principles of Data Management UCI, Fall 2018 Notes #09 External Sorting Instructor: Chen Li.
CS222: Principles of Data Management Lecture #10 External Sorting
General External Merge Sort
External Sorting.
CS222P: Principles of Data Management Lecture #10 External Sorting
CENG 351 Data Management and File Structures
Database Systems (資料庫系統)
External Sorting Chapter 13
CS222/CS122C: Principles of Data Management UCI, Fall 2018 Notes #09 External Sorting Instructor: Chen Li.
CSE 190D Database System Implementation
Presentation transcript:

6/27/20151 PSU’s CS External Sorting  Motivation  2-way External Sort: Memory, passes,cost  General External Sort: Memory, passes, cost  Optimizations  Snowplow  Double Buffering  Forecasting  Using a B+ tree index  Bucket Sort  Intergalactic Standard Reference  Graefe, Implementing Sorting in Database Systems 

6/27/20152 PSU’s CS 587 Learning Objectives  Derive formula for cost of external merge sort  Derive amount of memory needed to sort a file in 2 passes, using merge or bucket sort  Describe algorithm for generating longer initial runs and identify its best and worst cases  Describe forecasting and why it is useful  Identify when indexes should be used for sorting  Identify the pros and cons of external bucket vs merge sort.

6/27/20153 PSU’s CS 587 Why sort?  A classic problem in computer science!  Exercises many software and hardware features  Data is often requested in sorted order  e.g., find students in increasing gpa order  Sorting is first step in bulk loading B+ tree index.  Sorting useful for some query processing algoritms (Chapter 14)  Problem: sort 1Gb of data with 1Mb of RAM.  why not virtual memory?

6/27/20154 PSU’s CS 587 Sort algorithms?  If the data can fit in memory, which sort algorithm is best?  But most DBMS files will not fit in available memory  If the data is larger than memory, try the same alg.  Suppose for this data, your sort alg. requires 2 20 random memory accesses Memory access takes 1 microsec, disk takes 10 millisecs.  How much time is required to do your sort algorithm’s memory accesses? If there is enough memory to hold the data? If the data is four times the size of memory?

6/27/20155 PSU’s CS 587 External Sorts  Definition: When data is larger than memory.  An aside: What is “Memory”?  Physical memory? The DBMS is not the only player  We concluded that most in-memory sort algs won’t be effective for external sorting.  What sort algorithms are best for external sort?  Sort-based  Hash-based

6/27/20156 PSU’s CS Way External Merge Sort: Memory?  Pass 0: Read a page, sort it, write it.  How many buffer pages needed?  Pass 1, 2, 3, …, etc.:  How many buffer pages needed? Main memory buffers INPUT 1 INPUT 2 OUTPUT Result of Pass k+1 Result of Pass k 13. Sorting

6/27/20157 PSU’s CS Way External Merge Sort: Passes?  Assume file is N pages  Run = sorted subfile  What happens in pass Zero?  How many runs are produced?  What is the cost in I/Os?  What happens in pass 1?  What happens in pass i?  How many passes are required?  What is the total cost?

6/27/20158 PSU’s CS 587 Two-Way External Merge Sort: Cost  Each pass we read + write each page in file.  N pages in the file => the number of passes  So total cost is:  Idea: Divide and conquer: sort subfiles and merge Input file 1-page runs 2-page runs 4-page runs 8-page runs PASS 0 PASS 1 PASS 2 PASS 3 9 3,4 6,2 9,48,75,63,1 2 3,45,62,64,97,8 1,32 2,3 4,6 4,7 8,9 1,3 5,62 2,3 4,4 6,7 8,9 1,2 3,5 6 1,2 2,3 3,4 4,5 6,6 7,8 13. Sorting I/Os

6/27/20159 PSU’s CS 587 General External Merge Sort  To sort a file with N pages using B buffer pages:  Pass 0: use B buffer pages. How many sorted runs of B pages each are produced? Cost?  Pass 1,2, …, etc.: merge B-1 runs. How many runs are created after pass i? Cost of pass i? How many passes? Total Cost? B Main memory buffers INPUT 1 INPUT B-1 OUTPUT Disk INPUT 2... Disk... * Suppose we have more than 3 buffer pages. 13. Sorting

6/27/ PSU’s CS 587 Cost of External Merge Sort  Number of passes:  Cost = 2N * (# of passes)  E.g., with 5 buffer pages, to sort 108 page file  Number of passes is (1 +  log 4  108/5   ) = 4  Cost is 2(108)*4  Pass 0: Output is = 22 sorted runs of 5 pages each (last run is only 3 pages)  Pass 1: = 6 sorted runs of 20 pages each (last run is only 8 pages)  Pass 2: 2 sorted runs, 80 pages and 28 pages  Pass 3: Sorted file of 108 pages 13. Sorting

6/27/ PSU’s CS 587 How much memory is needed to sort a file in one or two passes?  N = number of data pages, B = memory pages available  To sort in One pass: N  B  To Sort in Two passes: 1+log B-1  N/B   2  N/B  (B-1) 1  Approximating B-1 by B, this yields   N  B  For example, if pages are 4KBytes, a 4GByte file can be sorted in Two Passes with ? buffers.

6/27/ PSU’s CS 587 Sorting in 2 passes: graphical proof  File is B pages wide  Each run is B pages  File is x pages high  Merge x runs in pass 1  x  B since we must merge x runs in B pages of memory  So N=xB  BB or  N  B The File, N pages Each run, 1xB pages B x

6/27/ PSU’s CS 587 Memory required for 2-pass sort N = # Pages in File [File size in Bytes] B = # Buffer Pages [ #Bytes] to sort in Two passes 2**10 [ 4Meg ]2**5 [128K] 2**20 [4 Gig]2**10 [ 4 Meg] 2**26 [256 Gig ]2**13 [32 Meg] Assuming page size of 4K

6/27/ PSU’s CS 587 Can we always sort in 1 or 2 passes?  Assume only one query is active, and there is at least 1 gig of physical RAM.  Yes: DB2,Oracle, SQLServer, MySQL  They allocate all available memory to queries  Tricky to manage memory allocation as queries need more memory during execution  No: Postgres  Memory allocated to each query, for sort and other purposes, is fixed by a config parameter.  Sort memory is typically a few meg, in case there may be many queries executing.

6/27/ PSU’s CS 587 Extremes of Sorting One Pass: N =B, 1-pass sort, cost = N NB Original Data Sorted Data Quick- sort Two Pass: N = B 2, 2-pass sort, cost = 3N N Original Data B Quicksort into B runs, length B 12..B12..B B B B Runs B Merge Sorted Data N

6/27/ PSU’s CS 587 Extreme Problems  Most sorting of large data falls between these two extremes  If we apply the intergalactic standard sort- merge algorithm, in every textbook, the cost for any dataset with B<N<B 2 will be 3M.  Must we always pay that large price?  Might there be an algorithm that is a hybrid of the two extremes?

6/27/ PSU’s CS 587 Hybrid Sort when N  3B  The key idea of hybrid sort is don’t waste memory.  Here is an example of the hybrid sort algorithm when N is approximately 3B. One + Pass: M  3B, 2-pass sort, cost = 3M – 2B Original Data B Quicksort into runs, length B, leaving the last run in memory BB Runs B Merge the runs on disk with the run in memory Sorted Data N

6/27/ PSU’s CS 587 Hybrid Sort in general  Let k = N/B  Arrange R so the last run is B-k pages  Cost is N + 2(N – (B-k)) = 3N -2B + 2(N/B) = 3N – 2( B-N/B)  When N=B, cost is N+1. When N=B 2, cost is 3N Original Data B Quicksort into runs, length B, leaving the last run in memory BB Runs B Merge the runs on disk with the run in memory Sorted Data N 12..K12..K 12..K12..K B-k pages

6/27/ PSU’s CS Maximizing initial runs  Defn: making initial runs as long as possible.  Why is it helpful to maximize initial runs?  If initial run size is doubled, what is the time savings?  How can you maximize initial runs?  What algorithm is best?

6/27/ PSU’s CS 587 Replacement Selection  Initialize empty priority queues CUR, NEXT  Read B buffer pages of data into CUR  Do  Pop record s with smallest key from CUR to current run  // key of s is now highest key in current run  If key of next input r >= key of s //Can put in current run insert r into CUR  else insert r into NEXT  If CUR is empty interchange NEXT and CUR and start a new run  Until (input is empty) 13. Sorting

6/27/ PSU’s CS 587 More on Replacement Selection  Cf. Knuth, vol 3 [442], page 255.  Theorem: average length of a run in replacement sort is 2B  Worst-Case:  What is min length of a run?  How does this arise?  Best-Case:  What is max length of a run?  How does this arise?  Quicksort is faster, but... 2B 13. Sorting

6/27/ PSU’s CS 587 How can we prove the Theorem?  Begin with some modeling assumptions  Data to be sorted are real numbers between 0 and 1  Data appear at a uniform rate and distribution  A snowplow picks up one datum as one falls  Picking up a datum == pop( ) off the queue  Each datum is infinitesimal  Each run begins when the plow passes zero 13. Sorting B

6/27/ PSU’s CS B 01 B 01 B 01 B 01 B 01 B CUR NEXT 13. Sorting 2B

6/27/ PSU’s CS 587 Snowplow: Conclusion  The figures on the previous page show that  At any time after run 0, the amount of snow = size of memory = B.  After the first run, the volume of snow removed in one circuit is 2B.  Cf. Larson and Graefe [471]  In spite of memory management problems, the snowplow optimization is very effective.

6/27/ PSU’s CS I/O for External Merge Sort  What else can we do to improve performance?  We have assumed I/O is done a page at a time  Text suggests reading a block of pages sequentially.  Pass 0: No problem  Pass 1,2,…: lowers fanin  Sometimes a win 13. Sorting

6/27/ PSU’s CS 587 External Sort’s jerky behavior  Recall that each input is one page  What happens after the last record on a page is output? B Main memory buffers INPUT 1 INPUT B-1 OUTPUT Disk INPUT 2... Disk...

6/27/ PSU’s CS 587 Double Buffering  To reduce wait time for I/O request to complete, can prefetch into `shadow block’.  This could increase the number of passes  In practice, most files still sorted in 1-2 passes. OUTPUT OUTPUT' Disk INPUT 1 INPUT k INPUT 2 INPUT 1' INPUT 2' INPUT k' block size b B main memory buffers, k-way merge 13. Sorting

6/27/ PSU’s CS 587 Forecasting  Cf. Knuth, vol 3, pg  Double Buffering requires Doubling memory  What a huge waste!  Most shadow buffers lie idle, unused, wasted.  How can we forecast which shadow buffers will be needed first?  Forecasting can achieve performance of double buffering with little memory A B C

6/27/ PSU’s CS 587 Sorting Records!  Sorting has become a blood sport!  Parallel sorting is the name of the game...  hmark hmark 13. Sorting

6/27/ PSU’s CS 587 Using B+ Trees for Sorting  Scenario: Table to be sorted has B+ tree index on sorting column(s).  Idea: Can retrieve records in order by traversing leaf pages.  Is this a good idea?  Cases to consider:  B+ tree is clustered Good idea!  B+ tree is not clustered Could be a very bad idea! 13. Sorting

6/27/ PSU’s CS 587 Clustered B+ Tree Used for Sorting  Cost: root to the left- most leaf, then retrieve all leaf pages (Alternative 1)  If Alternative 2 is used? Additional cost of retrieving data records: each page fetched just once. * Always better than external sorting! (Directs search) Data Records Index Data Entries ("Sequence set")

6/27/ PSU’s CS 587 Unclustered B+ Tree Used for Sorting  Alternative (2) for data entries; each data entry contains rid of a data record. In general, one I/O per data record! (Directs search) Data Records Index Data Entries ("Sequence set")

6/27/ PSU’s CS 587 Bucket Sort  Suppose search key values are 0-K  B pages in memory, N pages in the file  Pass 0: Partition the file into B-1 intervals  Inervals are not runs!  If the interval fits in one page, sort it B Main memory buffers OUTPUT 1 OUTPUT B-1 Disk OUTPUT 2... Disk... [0,K/(B-1)) [K/(B-1),2K/(B-1)) [(B-2)K/(B-1),K) INPUT

6/27/ PSU’s CS 587 Bucket sort cost  What happens after pass 0  ? intervals, each ? long  After pass 1?  ? intervals, each ? long  How much memory is required to sort in two passes?  Each interval is at most one page, or ?  Same as for external merge sort

6/27/ PSU’s CS 587 External Merge Sort vs External Bucket Sort  Approximately the same I/O cost  Same memory requirement for two passes  Same number of passes required to sort  Bucket sort has less CPU cost  Bucketizing is much cheaper than sorting/merging  But bucket sort is subject to skew  Thus merge sort is used in practice