1. 307: BW Performance Tuning - Queries / Data Loads
Ramesh Sampath Mike Maddox
Suresh Kandoor VJ Sudarsan

2. Objectives
To provide a comprehensive understanding of the factors affecting data load and query performance and to discuss the strategies to help identify, monitor and resolve performance issues. 2

3. Agenda Why Load Performance & Query Run-Times are important ?
Query Performance
Identifying long running Queries
Query tuning Techniques
Data Load Performance
Identifying long running loads
Improving Data Load times
Questions 3

4. Why Query Run-Times ? More Reporting Functions moving from R/3 to BW
User Frustration with Data Warehouse Promise
Impacts Number of Analysis Performed
User Productivity is affected by Query Run-Times
Quick response times, reduces Concurrent Users
Affects the bottom line !! 4

5. Query Performance Identifying and Isolating the slow Queries
Using BW Statistics
Using SAP Transactions
Exploring Possible Solutions to achieve better query run-times 5

6. Identifying long running Queries
Using BW Statistics
Average Run-Time of Query in Seconds 6

7. Identifying long running Queries
Using BW Statistics 7

8. Identifying long running Queries
Using BW Statistics 8

9. Identifying long running Queries
Using SAP Transactions
ST22 – Query Run Time exceeded set Limit for queries 9

10. Identifying long running Queries
Using SAP Transactions
Analyze SM50, SM66 to identify real-time query runs / users affected 10

11. Agenda Why Load Performance & Query Run-Times are important ?
Query Performance
Identifying long running Queries
Query tuning Techniques
Data Load Performance
Identifying long running loads
Improving Data Load times
Questions 11

12. Query Performance Query Tuning Techniques Analyze Query Plan
Identify degenerated indexes or incorrect statistics
Design Considerations (Data Modeling)
Queries on InfoCube and ODS
Selections on Hierarchies
Do’s and Don’ts in Query Building 12

13. Query Tuning Techniques
Analyze Query Plan (RSRT)
Check Read Mode of Queries
Displays List of Aggregates
applicable to query
based on selection
Displays Query Plan 13

14. Query Tuning Techniques
Specify Database Hints
Hints on Index Selection to observe the change to Execution Plan & Query Performance 14

15. Query Tuning Techniques
Specify Database Hints
Query Cost Estimate
Hints on Index Selection to observe the change to Execution Plan & Query Performance
Check the SQL Generated, to Identify Secondary Indexes on Dimension Tables, Master Data Tables 15

16. Query Tuning Techniques
Full Table Scan on Fact Table –
May lead to bad performance
Are the Statistics data Accurate ?
Hints on Index Selection to observe the change to Execution Plan & Query Performance 16

17. Query Tuning Techniques
Identify degenerated indexes or incorrect statistics: (RSRV)
Check DB Parameter Settings
Check Degenerated Indexes
Check Database Statistics 17

18. Query Tuning Techniques
Transaction: RSRV in BW 3.X
Check DB Parameter Settings 18

19. Query Tuning Techniques
Design Considerations for
Dimension Tables
Fact Tables
Aggregates
ODS Objects
Master Data 19

20. Query Tuning Techniques
Design Considerations for Dimension Tables:
Smaller Dimension Tables - Dimension Table size < 5% of Fact Table
Avoid M:N relationship
Logical grouping of objects in dimension
Fewer Dimension Tables per Cube
Avoid Near Line Item Dimensions
For large dimension tables, change index type to B-tree from Bitmap (default)
Create Secondary Indexes on Dimension tables 20

21. Query Tuning Techniques
Design Considerations for Fact Tables:
Partition large fact tables (over 10 million records)
Create Physical Partitioning (one Million records per partition)
Compression
Aggregates
Avoid Virtual Key Figures
Avoid exception aggregation 21

22. Query Tuning Techniques
Logical Partitioning of Cubes
Query against MultiProvider
Consolidated view of all data
MultiProvider
The Power of
Parallel Processing!
Parallel SELECT Statements
Partiton by Region
Europe
2000
America
2001
Asia
2002
Year 22
Sub-queries: Run against small structures (pruning)

23. Query Tuning Techniques
Design Considerations for Aggregates:
Data Selected from DB:Reported > 20:1
Best on Delta Capable Cubes
Use Nav. Attributes in Aggregate (Pro / Con). - Avoid adding frequently changing attributes
Create Aggregates on Hierarchy nodes
Create Base Aggregate and summarized Aggregate from Base
Avoid Virtual Key Figures / Virtual Characteristics
Avoid "Before Aggregation" Formulas in Queries
Split the delta capable data from full re-loads to create aggregates on delta capable cubes. 23

24. Query Tuning Techniques
Design Considerations for ODS Objects:
Create Secondary Indexes
Physical Partitioning using DB Tools
Locally managed indexes on Partitioned ODS
Use Multi-provider to split the Data
Create summarized cubes
High level analysis on Cubes, Detailed level analysis run queries or Jump to ODS queries.
Avoid Compounded Info Objects in ODS – Inefficient filtering by SAP
Index on Master Data P & Q Tables 24

25. Query Tuning Techniques
Design Considerations for Master Data:
Additional Over head with Time dependent Master data and hierarchies
Hierarchy vs. Navigational Attributes. Navigation attributes perform better than hierarchies.
Create additional index on the attributes of the master data that are used in query selections. X, Y Tables are used in Cube based Query filters. X, Y, P, Q Tables are used in ODS Based query filters. Create secondary indexes on Master data tables based on its usage 25

26. Query Tuning Techniques
Query Selection on Hierarchies
Temporary tables generated by SAP when querying based Hierarchy nodes.
Solution: Flatten the hierarchy nodes as attributes and select on them.
HIERNODE 26

27. Query Tuning Techniques
Do’s and Don’ts in Query Building:
Avoid generic queries
Break down queries by summary level & detail level analysis
Always have mandatory variables on time component
Take advantage of Partition Pruning
Limit the Use of hierarchies in Query Selections, use them extensively in Query Output
Use trailing Wild cards in query selections – Index!
Create indexes on MD, Dimension table objects used in query selections
Limit the use of ‘Before Aggregation’ Formulas
that provide High level Summarized information and detailed information 27

28. Query Tuning Techniques
Summary of Query Enhancing Techniques
Aggregates
Compression
Partitioning
Secondary Indexes
Accurate Database Statistics
Multi-Providers
Data Class / Table Space assignment for Cubes / ODS Objects and Indexes 28

29. Agenda Why Load Performance & Query Run-Times are important ?
Query Performance
Identifying long running Queries
Query tuning Techniques
Data Load Performance
Identifying long running loads
Improving Data Load times
Questions 29

30. Why Load Performance ?
With BW Evolving as Corporate Data Warehouse, more Functional Areas (Logistics / Financials / HR / …) are being serviced by BW
Small Window to Load Large Amounts of Data
Shrinking Data Load Batch Windows
Request for Frequent Data Loads. Middle of day data loads, hourly … 30

31. Data Load Performance Identifying and Isolating the Load Performance:
Using BW Statistics
Using SAP Standard Transactions
Exploring Possible Solutions to improve load run times 31

32. Identifying long running loads
BW Statistics – Load Times By Data Targets
Record Count
Load Time 32

33. Identifying long running loads
Detailed Analysis for Info Cube Loads
# of Records processed
Slow Update Rules
Inserts to Cube Slow 33

34. Identifying long running loads
BW Statistics – Load Times By InfoSource
Load Time
Record Count 34

35. Identifying long running loads
Detailed Analysis to identify cause on all Loads (Cube / ODS / Master Data)
Slow Transfer Rules
Slow Update Rules
# of Records Processed
Slow R/3 Extractor 35

36. Identifying long running loads
SM37 on BW system – ‘BI_ODSA*’ for ODS Activation Times SM50, SM66 – Process Overview ST04 – Database Analysis Overview 36

37. Identifying long running loads
Summary of factors affecting data loads
Inefficient Extract Programs
Inefficient logic in User Exits
Slow Data Transformation Services
Transfer Rules
Update Rules
Slow Data Loading Services
Loads into Info Cubes
Loads into ODS Objects 37

38. Agenda Why Load Performance & Query Run-Times are important ?
Query Performance
Identifying long running Queries
Query tuning Techniques
Data Load Performance
Identifying long running loads
Improving Data Load times
Questions 38

39. Improving Data Load Times
Strategies to improve Load Performance:
Extractor Performance
Info Cube Data Loads
Data Transformation Services (DTS)
ODS Data Loads
Master data Loads
Flat Files Data Loads 39

40. Improving Data Load Times
Strategies to improve Load Performance: (Contd.)
Perform Run-time Analysis on Extractor to identify Processing Times by ABAP Logic & Database Selects
Check Select Statements on Large R/3 Tables
Ensure that Selects are based on Primary Keys or secondary indexes
Check the usage of Run-time ABAP Memory by Internal Tables.
Schedule set up job in parallel 40

41. Improving Data Load Times
Every thing has been checked, but extract is still slow ?
Does the selection parameters entered at the Info package facilitate the use of Indexes !
Selection Parameters 41

42. Improving Data Load Times
Strategies to improve InfoCube Data Loads:
Load Master Data before Transactions to pre-create SID’s.
Number range Buffering on Large or Near Line item Dimensions for Large data loads
Packet Size – reduce the number of records per data packet
Secondary Indexes – Drop indexes on complete re-loads or loading significant amount of data (over 25% of existing cube data)
‘Turn-off’ Archival Logs before Large Initial Loads
On complete re-load of cube, do not delete dimension table entries, if you do not anticipate any changes to dimension entries. 42

43. Improving Data Load Times
Strategies to improve InfoCube Data Loads: (Contd.)
Incremental Data Loads
Do not drop Secondary Indexes on Incremental Loads
Do not select Refresh statistics after load 43

44. Improving Data Load Times
Strategies to improve DTS
Eliminate Single Selects on Update Routines & Transfer Routines
Utilize Start Routines
Avoid Transfer Rules to enhance Array Inserts
Utilize Start Routines
– Data Packet Level
Start Routines - Runs once per Data Packet. Loop in the Data packet and Update all relevant data fields at once.
There would still be scenarios where you would need to use Update Routines like Calculations, reversals etc. But they should not impact load performance.
Resist the temptation to write Select Statement on Transfer Routines
Depending on the size / indexes on the table you are selecting from they perform very bad since the select is executed per data record sent from source system
Data Integrity when writing DB Selects in Update / Transfer Rules. Are you Selecting from another ODS in Update Rules to ODS / Cubes ?.
In R/3, All related tables are updated in R/3 as a part of one LUW. Code at R/3 BW DataSource User exits would always get the correct information.
In BW, the data is loaded independently and the Routines should not impose dependency on data loads. This could lead to Data Integrity Problems in the Run-Maintain Environment.
Examine Update Routines
– Info Object Level 44

45. Improving Data Load Times
Strategies to improve ODS Data Loads
‘Turn-off’ Bex Reporting
Mark line item characteristics viz., doc Number as Attributes only
Large Initial Loads to ODS
‘Turn-off’ Database Archival Logs
Delete Secondary Indexes
First time load to ODS (ODS data is empty)
Load up to 1 Million Records, but do not exceed it.
Activate After the First load to facilitate Bulk Inserts
Adjust the Data packet size to be < 10 Data packets in the request. SAP has hard coded logic to check if data packet size is < 10% of the total request to process using Memory vs. single selects. Why this Difference ?
SAP does bulk processing vs. single record processing based on record counts when the ODS is empty.
, when ODS is not intended to be used in Bex Queries. Still use ODS in Infoset and in Multi-Providers through Infosets 45

46. Improving Data Load Times
Activating data in ODS Object
Setting ins SPRO 46

47. Improving Data Load Times
Strategies to improve subsequent data loads to ODS:
Activation – Single Record Inserts vs. Mass Insert when ODS is empty
Enable Unique records, if true. Bulk Inserts.
Data Packet Size (<10 Data Packets per Request)
Eliminate unused indexes.
Focus on smaller data loads for fewer records per activation to reduce the commit interval.
New in BW 3.X 47

48. Improving Data Load Times
Why does deletion of an Active Request from ODS takes a long time even for few thousand records ?
Data base partitioning on Active table
No Index on ODS Primary Key on ODS Change Log Table. Add a Index to Change log Table (SE11)
ODS Primary Key 48

49. Improving Data Load Times
Strategies to improve Master Data Loads:
Smaller Data Packet Size to minimize commit intervals
Convert full loads to delta extract based on date selection in the extract tables
Create accurate DB statistics
‘Turn-off’ Consistency Check.
Useful to select on data load errors to identify Row Numbers that have incorrect data, but has a performance impact 49

50. Improving Data Load Times
Strategies to improve Flat File Data Loads:
Use Logical File Names / Application Server Files
Run in Background Batch Mode
Break Large Files into Smaller Files and load them in parallel
Use Fixed Length Files in place of CSV files
SAP Converts CSV File inputs to Fixed Length before it starts processing the file. CSV File is better to view and analyze File data. Change to Fixed Length only if it improves performance significantly.
Maximize Parallel Loads
Increase the number of Batch Sessions at peak load times to run several data load jobs in parallel
Setup as Job triggered by Start and End events of Data loads or integrated into Data Load Process Chains 50

51. Improving Data Load Times
Summary of Data Load Enhancing Techniques
Extractor Performance on Source System
Info Cube Data Loads (Initial / Delta Loads)
Data Transformation Services (Transfer, Update Rules)
ODS Data Loads (Initial, Subsequent Loads)
Master Data Loads
Flat File Loads 51

52. Questions ? Questions Ramesh Sampath rsampath@codongroup.com
VJ Sudharsan
Mike Maddox
Suresh Kandoor
SAP Converts CSV File inputs to Fixed Length before it starts processing the file. CSV File is better to view and analyze File data. Change to Fixed Length only if it improves performance significantly.
Maximize Parallel Loads
Increase the number of Batch Sessions at peak load times to run several data load jobs in parallel
Setup as Job triggered by Start and End events of Data loads or integrated into Data Load Process Chains

53. Thank you for attending!
Please remember to complete and return your evaluation form following this session.
Session Code: [307]