Performance Evaluation of a DBMS Shahram Ghandeharizadeh Computer Science Department University of Southern California

Challenge n Imagine you develop the next super-duper buffer replacement technique, named SUPER. n How will you compare its performance with another technique, say LRU-K? – Using a scientific approach.

Scientific Evaluation n A controlled environment where only one variable is changing: – E.g., either SUPER or LRU-K is used. n The environment measures a metric of interest to compare the two techniques.

Step 1 n Decide what metric you want to emphasize when comparing SUPER with LRU-K. – Conceptual: Response time n Average person walking on Exposition will understand you as “saving time”. There is general consensus that time is money. – Physical: Cache hit ratio, Byte hit ratio n Average person will not understand you unless they have some understanding of a computer system and its hierarchical memory. n You need an apple-to-apple comparison

Step 2: Fix Environmental Variables n Identify a database and a workload to compare SUPER with LRU-K. n If a target application exists then: – Take a snap-shot of its database, – Write a sniffer to record the requests accessing your data. – Perform a trace-driven evaluation by playing back the recorded requests against LRU-K and SUPER managing your identified database. n Most often the target application does not exist.

Step 2 (Cont…) n If a target application does not exist then design a synthetic workload consisting of: – A database with a pre-specified number of records, n Fix-sized records numbered 1 to N. n Manipulate value of N to build small and large DBs. – And a workload generator that accesses these records. Must answer two key questions: n What data is accessed? n When is the data accessed?

What data to access? n Most often, certain data items are more popular than others. – rule: 90% of accesses reference 10% of data items. – rule: 80% of accesses reference 20% of data items. – Etc. n How? – Use a Zipfian distribution of access and manipulate the mean of the distribution.

Zipf Distribution

Motivation for Zipf n Zipf distributions have been shown to characterize use of words in a natural language (like English) and the popularity of library books, so typically – a language has a few words ("the", "and", etc.) that are used extremely often, and a library has a few books that everybody wants to borrow (current bestsellers) – a language has quite a lot of words ("dog", "house", etc.) that are used relatively much, and a library has a good number of books that many people want to borrow (crime novels and such) – a language has an abundance of words ("Zipf", "double-logarithmic", etc.) that are almost never used, and a library has piles and piles of books that are only checked our every few years (reference manuals for Apple II word processors, etc.)

Implementation of Zipf n Assuming C records are rank ordered based on their popularity: 1, 2, …, C. n With a Zipf-like distribution, the probablity of access to record j is a function of 1/power(j, u); see example in the next slide. n Value of u dictates the mean of the distribution. It is greater than or equal to zero and less than or equal to one. n X is 1 divided by the sum of power(j,u) for all clips. n Code is available from under software download.

Example of Zipf n Assume mean of distribution is 0.27 and the database consists of 3 records: 1, 2, and 3 – Record 1, power(1,0.27) = 1, 1/power(1,0.27) =1 – Record 2, power(2,0.27) = 1.21, 1/power(2,0.27)=0.83 – Record 3, power(3,0.27) = 1.35, 1/power(3,0.27)=0.74 n X = ( ) = 2.57 – Record 1, probability of access 1/2.57 = 0.39 – Record 2, probability of access 0.83/2.57 = 0.32 – Record 3, probability of access 0.74/2.57 = 0.29

When to Issue Requests? n It depends on the assumed simulation model: – Closed simulation models: n Consists of a fixed number of clients. n A client issues a request and does not issue another until its pending request is serviced. n The client may wait a pre-specified amount of time before issuing a new request. This delay is termed “think time”. – Open simulation models: n A process issues an average number of requests per unit of time. This is termed request arrival rate. It is denoted as lambda: λ

Closed Simulation Models Throughput SUPER LRU-K Multiprogramming level A technique (LRU-K) is better if it provides a higher throughput relative to the other technique as a function of the number of clients.

Open Simulation Models Average Response Time Arrival rate, λ SUPER LRU-K A technique (SUPER) is better if it supports a higher arrival rate without formation of queues. A slower technique results in queues with lower arrival rates.

Validate Your Simulation Model n How do you know your simulator has no bugs?

Validate Your Simulation Model n How do you know your simulator has no bugs? – Analytical models compute the response time of a system based on certain assumptions, e.g., a Poisson arrival rate. – Analytical assumptions are most often trivial: typically, they simplify the range of values allowed by a component of the simulation model. – Implement the assumptions and verify each component produces response times that match the analytical models. A divide-and-conquer approach.

Divide & Conquer n Validate components one at a time. – Example: how do you validate your implementation of a Zipfian reference pattern?

Divide & Conquer n Validate components one at a time. – Example: how to validate your implementation of a Zipfian reference pattern? 1. Assume a databases of few objects, say Use the analytical models to estimate the frequency of access to each object. 3. Run the Zipfian component of your simulator for a fixed number of requests (say 100,000) and measure the frequency of access to each object. 4. Items 2 and 3 should match very closely.

10 Commandments of Simulation Studies Shahram Ghandeharizadeh Computer Science Department University of Southern California

Outline n Why simulation studies? n 10 Commandments of simulation studies. n Review of each commandment. n Conclusions n References

Why simulation models? n To understand tradeoffs between alternative ways of implementing a functionality when: – A realistic implementation is too expensive/time-consuming. n Quantify the complexity of alternative implementations prior to a real system prototyping effort. Make sure it is not over-engineered: Many fold increase in software to get a 2% benefit? n Quantify tradeoffs with alternative strategies and choose the most appropriate one. n Gain insight to implement better strategies.

Why simulation models? n There are other reasons for developing a simulation model that falls beyond the scope of CSCI485: – To describe and understand systems too complex to explain, e.g., the human brain. – To identify factors/conditions with most impact on a performance metric, e.g., airplane crash in Singapore during a hurricane. Typically conducted using traces gathered from a real environment. – Capacity planning and risk analysis, e.g., design of power grids in anticipation of 20% increase in peak load in 2010, war-games.

Inherent limitation n A simulation is abstraction of a real system that may either exist or foreseen to exist in the future. n How do you know the abstraction level is correct?

Inherent limitation n A simulation is abstraction of a real system that may either exist or foreseen to exist in the future. n How do you know the abstraction level is correct? 1. We cannot know what we do not know. 2. We can never be sure that we have accounted for all aspects that could affect a simulation model’s ability to provide meaningful results. 3. Items 1 and 2 are specially true for future foreseen applications that do not exist today.

Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models

Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models n An implementation is specific and most often static. A simulation study can model a variety of possible system parameters, e.g., memory sizes in support of different buffer pool frames. More Abstract More Specific

Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models n A real prototype is more expensive to design, implement, test, and maintain. Cheaper Expensive

Alternatives to a simulation Real Prototypes/Implementation Simulation Study Analytical Models n Analytical models are more general, providing insights that might be overlooked by a narrow implementation. General Narrow

Components of a Simulator 1. Abstraction of a computing environment. A buffer pool with a fixed number of frames. 2. Implementation of alternative strategies. LFU, LRU, LRU-K 3. Abstractions of an application. – How requests are issued? Uniformly or in a bursty manner? – What processing is required from the system? – Key metric?

10 Commandments 1. Thou shall NOT obtain results from a computing environment unless its behavior is validated. 2. Thou shall NOT obtain results from an incorrect implementation of your strategies. 3. Thou shall NOT use unrealistic computing environments. 4. Thou shall NOT use unrealistic workloads or applications. 5. Thou shall NOT use someone else’s computing environment as the basis for your implementation unless you understand its abstractions and behavior for different parameter settings. 6. Thou shall NOT focus on absolute values. (Focus on the observed trends.)

10 Commandments (Cont…) 7. Thou shall NOT make a big deal out of less than 15% difference. Did the LRU-K paper show cache hit ratios higher than 15%? If not, how did it address this commandment? 8. Thou shall NOT report on a simulation with a few runs; establish statistical confidence. 9. Thou shall NOT publish results obtained from a simulator without disclosing all your assumptions. (Results must be reproducible by an independent scientist.) 10. Thou shall NOT perceive simulation as an end in itself. Validate your results against a real implementation. Techniques that appear feasible in a simulation study might be infeasible to implement.