OLAP Council APB-1 OLAP Benchmark Release II November 1998 Database Lab. Kwon,Sook-Kyo 3/30/2000

CONTENTS INTRODUCTION APPLICATION ENVIRONMENT DATABASE STRUCTURE DATA FILES QUERIES / QUERY DISTRIBUTION IMPLEMENTATION RULES FULL DISCLOSURE AND AUDITING

INTRODUCTION (1) OLAP Council Sponsor the development of an Analytical Processing Benchmark Successful OLAP Application must provide ‘Just-In-Time-Information’. Goal of APB-1 Measure a server’s overall OLAP performance rather than specific tasks. Include common business operations Bulk loading of data from internal or external data source Incremental loading of data from operational systems Aggregation of input level data along hierarchies Calculation of new data based on business models Time series analysis Queries with high degree of complexity Drill-down through hierarchies Ad hoc queries Multiple on-line session

INTRODUCTION (2) Judging Server’s Ability Represent complex business relationships Respond to changing business requirements Standard benchmark metric : AQM (Analytical Queries per Minute) Number of analytical queries processed per minute Data loading performance, calculation performance, query performance Calculated as Total time in seconds to perform the incremental data load + Total time in seconds to perform batch computation, if required + Total time in seconds to execute all queries = Total time in seconds for AQM measurements. AQM = Total number of queries executed * 60 / Total time in seconds for AQM measurement

Application Environment Contains a set of business operation that exercise basic functionality Sales and Marketing analysis system Synthesis of general business practices Analyze product sales to customers through distribution channels over time: unit sold, dollar sales, costs, margins are tracked by actual, budget, and forecast scenarios. Included items in the computation of AQM calculated values are preprocessed calculated values are stored in database calculated values at query time Updated incrementally at a specific time interval (weekly, or monthly) Not included items in the computation of AQM historical data loading

Database Structure (1) Six dimensions Product Time, scenario, measure, 3 aggregation dimensions that define database size (product, customer, and channel) Product 10 times the number of members in customer dimension minimum number of members : 10,000 7 levels : Top, Division, Line, Family, Group, Class, Code Customer 100 times the number of members in channel dimension minimum number of members : 1,000 3 levels : Top, Retailer, Store

Database Structure (2) Channel fewest members Time minimum number of members : 10 number of members in channel is input parameter to GEN program 2 levels : Top, Base Time dimension : two years (1995 and 1996) of monthly members Julian calendar is used hierarchy : quarterly, yearly, year-to-date aggregations current month in the benchmark : June, 1996 aggregation : summation except; Inventory - ending balance calculation Cost factor measures - average balance Percentage measure - cannot be aggregated

Database Structure (3) Scenario 3 base values : actuals and budget (from data files), forecast (modeled scenario ) Variance values : Budget vs. Actuals, Forecast vs, Actuals Measures Input Unit sold varies by product,customer, channel, time, scenario Dollar Sales varies by product, customer, channel, time, scenario Inventory varies by product, customer, time Product Cost varies by product, time, scenario Shipping Cost varies by customer, time, scenario Calculated Average Price = Dollar Sales /Unit Sold Cost, Margin, Margin Percent, Smoothed Sales

Data Files (1) Two set of data files Initialize the database, incremental processing APB1GEN program generates all data files Files area presented in both tree and level formats Product Hierarchy - Level Format Tree Format Column Width Filed Column Width Field 1 12 Code 1 12 Member 13 12 Class 13 12 Parent 25 12 Group 25 12 Level 37 12 Family 49 12 Line 61 12 Division

Data Files (2) Hierarchy data files : product, customer, channel Reference scenario members by the names : ACTUAL, BUDGET, FORECAST Reference members of time dimension at the month level Not in any sorted order Sorting time will be included in the calculation of AQM

Queries / Query Distribution Ad hoc and very dynamic The time periods, products, customers, and channels must be dynamically generated from their respective hierarchies. To best simulate a realistic operating environment, each of the queries will have substitution parameters. Query 1 - Channel Sales Analysis : 10 % Query 2 - Customer Margin Analysis : 10 % Query3 - Product Inventory Analysis : 15 % Query 4 - Time Series Analysis : 3 % Query 5 - Customer Budget : 5 % Query 6 - Product Budget : 5 % Query 7 - Forecast Analysis : 15 % Query 8 -Budget performance : 20 % Query 9 - Forecast Performance : 15 % Query 10 - Ad Hoc : 2 %

Implementation Rules (1) Software used to implement Must be generally available Should not be engineered specially Use of alpha, beta, limited release, non full-production is not accepted. Database design restriction All data must be stored on the server All calculation must be performed on the server APB-1 is executed in six steps: 1. Execution of APB.EXE program to produce hierarchy / historical data 2. Database initialization and historical data load and optional pre-calculation 3. Execution of the APB.EXE program to produce incremental data files 4. Incremental data load and optional pre-calculation 5. Execution of APB.EXE program to produce query data files 6. Query execution

Implementation Rules (2) All processing must be done on the server. The execution time for step begins when the first user initiates the first query and ends when the last user ends the last query. The results of each query must be written to disk on the client platform to be available for auditing.

Full Disclosure and Auditing (1) Full disclosure is required by anyone publishing APB-1 benchmark results so that others can duplicate the results same documentation, settings, products and environment. The disclosure includes the information ; an evaluator needs to judge a server’s processing performance its suitability to the task of analytical processing Disclosure Items; Audit Report, Database schema, Any code, scripts, programs and / or interactive user processes used for the dat loading…….. Total number of users, data file size, AQM metric, OLAP software Server / Client hardware, network S/W H/W configuration...

Full Disclosure and Auditing (2) The audit of the APB-1 benchmark will consist of the following six tasks : . Verifying that benchmark was run in the prescribed order as detailed in the implementation rules section . Examining the programming code on both the client and the server to verify that : The client’s sole role is to pass the queries to and receive the results from the server The server performs all calculations

Full Disclosure and Auditing (3) . Verifying that the results returned are correctly calculated. Thirty queries (three of each query type) must be randomly chosen and must be audited to Ensure data accuracy Determine that the dimensions members returned are correct . Ascertaining that all disclosure items are correctly stated in the disclosure document. . Verifying that the benchmark implementation conforms completely to the OLAP council APB-1 benchmark specification, addenda and errata. Auditor must also specify which revision of the APB-1 benchmark specification and which version of APB.EXE program was used. .Producing a report detailing the audit steps and results.