Increase Your Productivity by Doing Less Next Presentation: Increase Your Productivity by Doing Less Presenter: Arthur Tabachneck Art holds a PhD from Michigan State University, has been a SAS user since 1974, is president of the Toronto Area SAS Society and has received such recognitions as the SAS Customer Value Award (2008), SAS-L Hall of Fame (2011), SAS Circle of Excellence (2012) and, in 2013, was recognized as being the first SAS Discussion Forum participant to be awarded more than 10,000 points Copyright © 2010, SAS Institute Inc. All rights reserved. 1

Increase Your Productivity by Doing Less Arthur Tabachneck Thornhill, ON Canada Xia Ke Shan Beijing, China Joe Whitehurst Robert Virgile Atlanta, GA Lexington, MA Copyright © 2010, SAS Institute Inc. All rights reserved.

our Truth in Advertising commitment WARNING: The paper I'm about to present: was originally a minor subset of a major paper I presented in the Beyond the Basics section it was written because we thought of the idea for the original paper too late to submit to SGF2013 but, due to a cancellation, SGF had an opening for a 10 minute Coder's Corner paper but, while it is clearly a disturbing example of extreme opportunism, it may be the most significant 10 minute paper/presentation I've had the privilege of co-authoring #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 3

What the %transpose() macro is It's a SAS macro Looks and feels like PROC TRANSPOSE Has all of the PROC TRANSPOSE options and statements + two additional ones Can run between 9 and 12 as much as 200 times or more faster than using PROC SORT and PROC TRANSPOSE #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 4

Have you ever had to flip a SAS dataset from being tall to being wide? i.e., from: idnum date var1 1 2001JAN SD 2001FEB EF 2001MAR HK 2 GH 2001APR MM 2001MAY JH #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 5

flipping a SAS dataset to: idnum var1_2001JAN var1_2001FEB var1_2001MAR var1_2001APR var1_2001MAY 1 SD EF HK   2 GH MM JH #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 6

if you have, you are probably already familiar with PROC TRANSPOSE Note to self: remember to first sort the data proc transpose data=have out=want (drop=_:) prefix=var1_; by idnum; var var1; id date; run; Not difficult, but you need to know which are options which are statements and, if needed, remember to sort your data before hand #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 7

How many of you have ever: forgotten to run proc sort before running another proc that required sorted data? run proc sort but didn't include the options that can make the process more efficient (e.g., noequals, presorted and tagsort)? run a proc that only used a few of a file's variables, but didn't include a keep dataset option to limit the amount of data that had to be processed? #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 8

Compare the performance of the following two sets of almost identical code run on a file with 40,000 records and 1,002 variables PROC SORT data=have out=need; by idnum date; run; took 2.41 seconds CPU time PROC TRANSPOSE data=need out=want (drop=_:) prefix=var1_Qtr; by idnum; var var1; id date; format date Qtr1.; run; took 0.74 seconds CPU time #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 9

Compare the performance of the following two sets of almost identical code run on a file with 40,000 records and 1,002 variables PROC SORT data=have (keep=idnum date var1) out=need noequals; by idnum date; run; took 0.33 seconds CPU time PROC TRANSPOSE data=need out=want (drop=_:) prefix=var1_Qtr; by idnum; var var1; id date; format date Qtr1.; run; took 0.16 seconds CPU time 6.39 times faster #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 10

More importantly compare the real time performance of running the two versions of PROC SORT PROC SORT data=have out=need; by idnum date; run; took 109.54 seconds real time PROC SORT data=have (keep=idnum date var1) out=need noequals; by idnum date; run; took 0.57 seconds real time 192.2 times faster #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 11

You can achieve the same performance gains by running the following code %transpose(data=have, out=want (drop=_:), prefix=var1_Qtr, by=idnum, var=var1, id=date, sort=yes, format=date Qtr1.) The macro's benefits less code thus less chance for error same efficiency as the optimized code no need to know which PROC TRANSPOSE features are options and which are statements further code reductions possible by setting common defaults for the named parameters #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 12

How the %transpose() macro works First, all of the parameters are declared: %macro transpose( data=, out=, var=, prefix=, suffix=, let=, name=, label=, sort_options=, id=, sort=, idlabel=, delimiter=, by=, copy=, format=); the same options and statements you would use with PROC TRANSPOSE + two new named parameters (sort and sort_options) #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 13

If &var is null populate it with all numeric variables data _temp; set &data. (obs=1 drop=&by. &id. &copy.); run; %if %length(&var.) eq 0 %then %do; proc sql noprint; select name into :var separated by " " from dictionary.columns where libname="WORK" and memname="_TEMP" and type="num" ; quit; %end; #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 14

If the sort parameter eq 'yes' run proc sort %if %sysfunc(upcase("&sort.")) eq "YES" %then %do; proc sort data=&data out=_temp by &by. &id.; run; %let data=_temp; %end; (keep=&by. &id. &var. &copy.) &sort_options. noequals; #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 15

using all of the specified parameters Run PROC TRANSPOSE using all of the specified parameters proc transpose data=&data. (keep=&by. &id. &var. &copy.) %if %length(&delimiter.) gt 0 %then delimiter=&delimiter.; %if %length(&label.) gt 0 %then label=&label.; %if %length(&let.) gt 0 %then let; %if %length(&name.) gt 0 %then name=&name.; %if %length(&out.) gt 0 %then out=&out.; %if %length(&prefix) gt 0 %then prefix=&prefix.; %if %length(&suffix) gt 0 %then suffix=&suffix.; ; %if %length(&by.) gt 0 %then by &by.;; %if %length(&copy.) gt 0 %then copy &copy.;; %if %length(&id.) gt 0 %then id &id.;; %if %length(&idlabel.) gt 0 %then idlabel &idlabel.;; %if %length(&var.) gt 0 %then var &var.;; %if %length(&format.) gt 0 %then format &format.;; run; #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 16

finally, delete temporary file proc delete data=work._temp; run; %mend transpose; #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 17

Potential Applications transposing files quicker using the macro as a template for creating similar macros which incorporate sort and efficiency options for other SAS procs that could benefit from the approach #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 18

Presentation Overview How this paper came about  What the %transpose macro is  The macro's benefits  How the macro works  Potential applications of the macro  #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 19

Questions? #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved. 20

Your comments and questions are valued and encouraged Contact the Authors Arthur Tabachneck, Ph.D. President, myQNA, Inc. Thornhill, ON art297@rogers.com code name: art297 Joe Whitehurst High Impact Technologies Atlanta, GA joewhitehurst@gmail.com code name: joe whitehurst Robert Virgile Robert Virgile Associates, Inc. Lexington, MA rvirgile@verizon.net code name: astounding Xia Ke Shan Beijing, China keshan.xia@gmail.com code name: ksharp #SASGF11 Copyright © 2010, SAS Institute Inc. All rights reserved.