1. SAS/ACCESS Performance Initiatives PDXSUG 2010 Howard Plemmons Senior Software Manager SAS Howard.Plemmons@sas.com 919-531-7779 Copyright © 2010 SAS Institute Inc. All rights reserved.

2. Introduction Overview SAS/ACCESS Performance - software, practice and process Performance Initiatives - no specific priority order Performance Enhancements in SAS 9.2M3 Q&A PDXSUG 2010

3. ACCESS – Big Picture DBMS Server DBMS Server Table DBMS Client DBMS Client Table SAS Procedure SAS Data Step, etc. SAS ACCESS PDXSUG 2010

5. 10 – readbuff SAS Technical Support DB2 UNIX/PC, DB2 z/OS, HP Neoview, Microsoft SQL Server, ODBC, OLE/DB, Oracle, Sybase PDXSUG 2010

6. Number 10 - readbuff Definition Specification Performance Implication PDXSUG 2010

7. Number 10 - readbuff libname x … readbuff=1000; data a; set x.bigtab(readbuff=20); run; PDXSUG 2010

8. Number 10 - readbuff Memory consumption Diminishing returns Defaults PDXSUG 2010

9. Number 10 - readbuff data _null_; set lib.data(readbuff=xxx); run; PDXSUG 2010

10. 9 – bulkload SAS Technical Support DB2 UNIX/PC, DB2 z/OS, HP Neoview, Netezza, ODBC, OLE DB, Oracle, Sybase, Teradata PDXSUG 2010

11. Number 9 - bulkload Key to DBMS bulk loaders Option controlled – BL_ Performance Implications PDXSUG 2010

12. Number 9 - bulkload Sample DB2 bulkload code /*--- point to data ---*/ libname dbms db2 …; libname sas …; /*--- load SAS data into DB2 ---*/ proc sql; create table dbms.ldata (bulkload=yes bl_options=‘norowwarnings’) as select * from sas.sas_data; quit; PDXSUG 2010

13. Number 9 - bulkload Bulk Loading Options GeneralIMPORTCLILOAD BL_CODEPAGE BL_DATAFILE BL_DELETE_DATAFILE BL_LOG BL_OPTIONS BL_REMOTE_FILE BL_SERVER_DATAFILE BL_WARNING_COUNT BL_CODEPAGE BL_DATAFILE BL_DELETE_DATAFILE BL_LOG BL_OPTIONS BL_ALLOW_READ_ACCESS BL_ALLOW_WRITE_ACCESS BL_COPY_LOCATION BL_CPU_PARALLELISM BL_DATA_BUFFER_SIZE BL_DISK_PARALLELISM BL_EXCEPTION BL_INDEXING_MODE BL_LOAD_REPLACE BL_LOG BL_METHOD BL_OPTIONS BL_RECOVERABLE BL_REMOTE_FILE PDXSUG 2010

14. 8 – implicit passthru updates R&D PDXSUG 2010

15. Number 8 – implicit passthru updates IP (implicit passthru) - 9.1 and 9.2 Implications PDXSUG 2010

16. Number 8 – implicit passthru updates SQL with issues /*--- point to data ---*/ libname x …; proc sql; create table work.a as select * from (select c.dept, (select num as newnum from x.tabs01 where newnum=1) as col2 form x.tabs03 c) qu1, x.tabs04 d where qu1.col2 = d.empnum and qu1.col2 > 0; PDXSUG 2010

17. Number 8 – implicit passthru updates SQL pushed to the database today – ERROR: ORACLE prepare error: ORA-00936: missing expression. SQL statement: select c."DEPT", ( select TABS01."NUM" as newnum from TABS01 where TABS01."NUM" = 1 ) as col2 from TABS03 c where ( select from TABS01 ) > 0. ACCESS ENGINE: SQL statement was not passed to the DBMS, SAS will do the processing. SELECT NULL as TABLE_QUALIFIER, NULL as TABLE_OWNER, t.table_name as TABLE_NAME, t.num_rows as ROW_COUNT FROM user_tables t WHERE t.table_name='TABS03‘ SELECT NULL as TABLE_QUALIFIER, NULL as TABLE_OWNER, t.table_name as TABLE_NAME, t.num_rows as ROW_COUNT FROM user_tables t WHERE t.table_name='TABS04‘ SELECT "EMPNUM", "STATE" FROM TABS04 WHERE ("EMPNUM" > 0 ) SELECT "DEPT" FROM TABS03 SELECT "NUM" FROM TABS01 WHERE ("NUM" = 1 ) PDXSUG 2010

18. Number 8 – implicit passthru updates SQL pushed to the database – select qu1.”DEPT”, qu1.COL2, d.”EMPNUM”, d.”STATE” from (select c.”DEPT”( select TABS01.”NUM” as NEWNUM from TABS01 where TABS01.”NUM” = 1) as COL2 from TABS03 c) qu1, TABS04 d where (qu1.COL2 = d.”EMPNUM”) and (qu1.COL2>0) Database does the processing PDXSUG 2010

19. 7 – put processing R&D PDXSUG 2010

20. Number 7 – put processing SAS Formats DBMS Processing Examples PDXSUG 2010

21. Number 7 – put processing SAS Format – user defined proc format; value fastcar 1-45 = ‘slow’ 46-75 = ‘lawful’ 76-90 = ‘commuter’ 91-125 = ‘ticket’; run; PDXSUG 2010

22. Number 7 – put processing Using formats in SQL libname x …; proc sql; select name, address, phone, speed from x.baddriver where put(speed, fastcar.) = ‘ticket’; run; PDXSUG 2010

23. Number 7 – put processing SQL pushed down for processing SELECT "NAME", "ADDRESS", "PHONE", "SPEED" FROM BADDRIVER WHERE ( ( ("SPEED" BETWEEN 91 AND 125 ) ) … Before PUT processing SELECT "NAME", "ADDRESS", "PHONE", "SPEED" FROM BADDRIVER PDXSUG 2010

24. 6 – utilities R&D Teradata PDXSUG 2010

25. Number 6 - utilities Other than bulkload Performance impact Usage PDXSUG 2010

26. Number 6 - utilities Usage – Teradata Multiload and FastExport libname x teradata …; proc append data=a base=x.mytd_tab( multiload=yes); libname y teradata … fastexport=yes; /*--tpt-- */ data a; set y.mytd_tab; run; PDXSUG 2010

27. Number 6 - utilities GeneralMULTILOAD FASTEXPORT MBUFSIZE ML_CHECKPOINT ML_ERROR1 ML_ERROR2 ML_LOG ML_RESTART SLEEP TENACITY PDXSUG 2010

28. 5 – data step modify SAS Technical Support DB2 – Z/OS PDXSUG 2010

29. Number 5 – data step modify New? Efficiencies in SAS processing Example PDXSUG 2010

30. Number 5 – data step modify libname dblib db2 …; data dblib.employee; modify dblib.employee upd; by empid; select (_iorc_ ); when(%sysrc(_sok)) replace; when(%sysrc(_dsenmr)) do; _error_ =0; put ‘No match was found for’empid=; end; otherwise do; put ‘Unexpected error’; put _all_; end; run; PDXSUG 2010

31. Number 5 – data step modify Code example – SQL process libname dblib …; proc sql; update dblib.employee a set name=(select name from upd b where a.empid=b.empid), dept=(select dept from upd b where a.empid=b.empid), salary=(select salary from upd b where a.empid=b.empid) where empid in (select empid from upd); quit; PDXSUG 2010

32. 4 – functions R&D DB2 UNIX/PC, DB2 z/OS, HP Neoview, Informix, MySQL, Netezza, ODBC, OLE DB, Oracle, Sybase, Teradata PDXSUG 2010

33. Number 4 - functions SAS -> DBMS Implementation Examples PDXSUG 2010

34. Number 4 - functions ACCESS engine dictionary libname x oracle … sql_functions_copy=saslog; SAS Function Mappings provided by SAS ACCESS engine: SAS DBMS FUNCTION NAME ------------- ABS ABS ARCOS ACOS … UPCASE UPPER … PDXSUG 2010

35. Number 4 - functions Invoking SQL functions libname x oracle … sql_functions = <ALL | “EXTERNAL_APPEND = ” | “EXTERNAL_REPLACE = ”> ACCESS Engine Function File PDXSUG 2010

36. Number 4 - functions Sample SAS code – pushdown 100% libname x oracle …; proc sql; select name, address, phone from x.clients where upcase (country) = ‘USA’; quit; UPPER CLIENTS PDXSUG 2010

37. 3 – sql process R&D PDXSUG 2010

38. Number 3 – sql process Pushing SQL through IP Importance Examples PDXSUG 2010

39. Number 3 – sql process Examples libname x …; /*--- control pushdown – default nodbidirectexec ---*/ options dbidirectexec; proc sql; create table x.newtab as select * from x.oldtab; delete * from x.newtab where custid > 1010; quit; PDXSUG 2010

40. Number 3 – sql process Examples create table x.newtab as select * from x.oldtab; With DBIDIRECTEXEC ORACLE_433: Executed: on connection 3 CREATE TABLE new as ( select old."X", old."Y" from OLD ) Without DBIDIRECTEXEC (NODBIDIRECTEXEC) ORACLE_420: Executed: on connection 4 CREATE TABLE NEW(X NUMBER,Y VARCHAR2 (3)) ORACLE_421: Prepared: on connection 4 INSERT INTO NEW (X,Y) VALUES (:X,:Y) ORACLE_422: Executed: on connection 4 INSERT statement ORACLE_421 … PDXSUG 2010

41. 2 – in-database R&D Teradata PDXSUG 2010

42. Number 2 – in-database procedures PROC FREQ in Teradata In-database concept Examples PDXSUG 2010

43. Number 2 – in-database procedures Example options sqlgeneration=DBMS; libname x teradata …; proc freq; data=x.tdtab; run; ( select * from tdtab) vs. ( in database SQL) PDXSUG 2010

44. Number 2 – in-database procedures Some gotcha’s and fallback First obs and rename as Data set options Weight variables as analysis variables … PDXSUG 2010

45. Other In-Database Procedures Means Summary Frequency Rank Etc. PDXSUG 2010

46. 1 – examine, measure, modify SAS Technical Support /R&D PDXSUG 2010

47. Number 1 – examine, measure and modify Performance analysis - why Application to your conditions Options that may Help PDXSUG 2010

48. Number 1 – examine, measure and modify Options that may Help options sastrace = ‘,,,d’ nostsuffix sastraceloc=saslog; options sastrace = ‘,,,s’ nostsuffix sastraceloc=saslog; SAS Technical Support PDXSUG 2010

49. Number 1 – examine, measure and modify Debug output 3 data a; 4 set x.dept; run; NOTE: There were 4 observations read from the data set X.DEPT. Summary Statistics for ORACLE are: Total row fetch seconds were: 0.000631 Total SQL execution seconds were: 0.036121 Total SQL prepare seconds were: 0.028397 Total seconds used by the ORACLE ACCESS engine were 0.429114 PDXSUG 2010

50. Existing Products - General Commitment to SAS customers Release Rule of Thumb – what is supported Special Situations – SAS/DBMS client combos DB2 z/OS PC Files PDXSUG 2010

51. Existing Products – DB2 Message Improvements Old – Warning: File X.MYTABLE.DATA does not exist Warning: Table X.MYTABLE has not been dropped New (following appears before the warnings above) – DB2 ERROR: RESULT OF SQL STATEMENT: DSN00204E MYID.MYTABLE IS AN UNDEFINED NAME A DUMP OF THE SQLCA FOR THE GET DIAGNOSTICS REQUEST FOLLOWS: ROW NUMBER:0 ERROR CONDITION: 1 REASON CODE: 0 SQLCODE: -204 SQLSTATE: 42704 SQLERRP: DSNXOTL PDXSUG 2010

52. Existing Products – DB2 Performance/Usage Improvements READBUFF Extended SQL Function Support Database Read-Only Access Support DB2 Explain Support PDXSUG 2010

53. Existing Products – DB2 Performance Improvements – Readbuff (65%) /*--- readbuff=1 Data Set Option ---*/ Summary Statistics for DB2 are: Total row fetch seconds were: 23.888685 Total SQL prepare seconds were: 0.0108044 Total seconds used by the DB2 ACCESS engine were:45.749788 /*--- readbuff=10000 Data Set Option ---*/ Summary Statistics for DB2 are: Total row fetch seconds were: 8.278659 Total SQL prepare seconds were: 0.000330 Total seconds used by the DB2 ACCESS engine were:28.347795 PDXSUG 2010

54. Existing Products – DB2 Performance Improvements – Extended SQL Function Support libname x db2 [ssid= ] SQL_FUNCTIONS_COPY=SASLOG; proc sql; create table a as select substr(address, 1, 10) from x.my_db2_tab; quit; PDXSUG 2010

55. Existing Products – DB2 Usage Improvement – Read Only ACCESS DB2UPD run-time option (default YES) DB2UPD=NO blocks update access to tables does not affect DB2 security settings PDXSUG 2010

56. Existing Products – DB2 Performance Improvements - Explain New settings for SASTRACE, explain and explain and execute – options sastrace=‘,,,e’; options sastrace=‘,,,x’; Output – DB2 NOTE: The QUERYNO for the EXPLAIN is 2033622 PDXSUG 2010

57. Existing Products – PC Files Server - Overview 32- and 64-Bit Windows Microsoft Excel Microsoft ACCESS ACE JET ODBC JMP ® STATA® SPSS® SAS/ACCESS Interface to PC File Formats (UNIX) SAS/ACCESS Interface to PC File Formats (Windows) PC Files Server PDXSUG 2010

58. Existing Products – PC Files Server PC Files Server – Performance Improvements Microsoft Excel20 Columns50 Columns200 Columns Import 1,000 Rows2X3X4X Import 10,000 Rows4X Import 65,000 Rows3X Export 1,000 Rows7X 4X Export 10,000 Rows11X9X4X Export 65,000 Rows6X5X4X PDXSUG 2010

59. Existing Products – PC Files Server PC Files Server – Performance Improvements Microsoft Access data imports and exports based on improvements in the third maintenance release for 9.2:Microsoft Access 20 Columns50 Columns 200 Columns Import 1,000 Rows2X4XACE Driver Limitation Import 10,000 Rows9X7XACE Driver Limitation Import 65,000 Rows9X8XACE Driver Limitation Export 1,000 Rows13X14XACE Driver Limitation Export 10,000 Rows37X38XACE Driver Limitation Export 65,000 Rows33X ACE Driver Limitation PDXSUG 2010

60. Existing Products – PC Files Server PC Files Server – From UNIX / Windows proc import dbms=pcfs datafile=“c:\db.jmp” out=a replace; server=fs; run; proc import dbms=excelcs datafile=“c:\db.xls” out=b replace; server=fs; run; proc export dbms=pcfs outfile=“c:\newjmp.jmp” data=a replace; server=fs; run; PDXSUG 2010

61. Summary Improvements using options Improvements using features Improvements using processes Improvements in SAS 9.2 PDXSUG 2010

62. Q&A PDXSUG 2010