1. Copyright © 20088 SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Useful by Association: Adding Practical Value by Analyzing SAS ® 9 Data Integration Metadata Steve Morton, Applied System Knowledge Ltd, Henley-on-Thames, U.K.

2. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Useful by Association:… Overview of the Paper SAS 9 Metadata - the what and how Basics of the SAS Open Metadata Architecture Reading metadata in SAS ® DATA step Simple Attribute values Listing all Objects of a given type Analysis using Associations Developing analysis utilities Example: creating an analysis program to list all input & output tables for every Job in a repository Conclusions

3. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Why the DATA step?  Documentation: SAS ® Open Metadata Interface: Reference Documents the structure & concepts Gives examples – mostly in Java, C++ or Visual Basic Also describes DATA step functions On SAS projects you can always find SAS programmers! You can’t necessarily rely on having access to Java, C++ or VB expertise

4. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Getting Started… Main features of interest: Objects −The ‘things’ such as Table, Column, Job Attributes −Properties of Objects such as Name, Type, Length Associations −Connection between Objects, such as Table-to-Columns Metadata Browser in the SAS desktop environment SAS language features …

5. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. SAS Language Features for Metadata Access PROC METADATA XML query structure as input XML output – requires XSL template to read Verbose and “un-SAS-like” DATA step functions METADATA_GETNOBJ, METADATA_GETNATR and so on Input name text and/or Universal Resource Identifiers Output text, URIs, return codes Very natural to SAS programmers!

6. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Function: metadata_getnatr Get N th attribute of an object: rc = metadata_getnatr( "omsobj:SASLibrary?@Name = 'sastest'", N, attribute, value); Input URI and attribute number Output attribute name & value

7. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Program: read all attributes for a SASLibrary data _null_; a = 0; rc = 0; length attr value $256; Put 'Attributes of sastest library:'; do while(rc >= 0); A + 1; rc = metadata_getnatr( "omsobj:SASLibrary?@Name = 'sastest'", A, attr, value); if value ne '' then do; put +5 attr ' = ' value; end; stop; run;

8. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Output: read all attributes for a SASLibrary Attributes of sastest library: Name = sastest MetadataUpdated = 05Nov2007:18:40:11 MetadataCreated = 05Nov2007:18:40:11 Libref = sastest IsPreassigned = 0 IsDBMSLibname = 0 Engine = BASE Id = A5R7K9XX.BH0000RT

9. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Library Attributes in Metadata Browser

10. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other Useful Functions Metadata_Getnobj Search for N th object that matches a search URI Use to get all objects of a type, or partial-name match Metadata_Getattr Get specific attribute value by name Metadata_Getnasl Get each association name than can exist for an object Metadata_Getnasn Get N th association URI

11. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Program: read all Associations for a SAS Library data _null_; length id $20 type attr prop assoc value lib_uri auri $256; nobj = metadata_getnobj("omsobj:SASLibrary?@Name = 'sastest'", 1, lib_uri); if nobj gt 0 then do; put lib_uri=; n = 0; rc = 0; Put 'Associations of sastest library:'; do while(rc >= 0); n + 1; rc = metadata_getnasl(lib_uri, n, assoc); if assoc ne '' then do; i = 0; arc = 0; do while(arc >= 0); i + 1; arc = metadata_getnasn(lib_uri, assoc, i, auri); if arc >= 0 then do; put +5 assoc i ': ' auri=; end; *if arc…; end; *do while…; end; * if assoc…; end; *do while…; end; *if nobj…; stop; run;

12. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Output: all Associations for a SAS Library lib_uri=OMSOBJ:SASLibrary\A5R7K9XX.BH0000RT Associations of sastest library: DeployedComponents 1 : auri=OMSOBJ:ServerContext\A5R7K9XX.AT000001 Tables 1 : auri=OMSOBJ:PhysicalTable\A5R7K9XX.BJ000006 Trees 1 : auri=OMSOBJ:Tree\A5R7K9XX.AK0002BE UsingPackages 1 : auri=OMSOBJ:Directory\A5R7K9XX.B00000RT UsingPrototype 1 : auri=OMSOBJ:Prototype\A5R7K9XX.AA00003 lib_uri=OMSOBJ:SASLibrary\A5R7K9XX.BH0000RT Associations of sastest library: DeployedComponents 1 : auri=OMSOBJ:ServerContext\A5R7K9XX.AT000001 Tables 1 : auri=OMSOBJ:PhysicalTable\A5R7K9XX.BJ000006 Trees 1 : auri=OMSOBJ:Tree\A5R7K9XX.AK0002BE UsingPackages 1 : auri=OMSOBJ:Directory\A5R7K9XX.B00000RT UsingPrototype 1 : auri=OMSOBJ:Prototype\A5R7K9XX.AA00003 PhysicalTable  tables of the library Tree  custom folder in which the library is found Directory  directory path of the data

13. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Getting directory path details – key statements  Search for all SASLibrary objects: lrc = metadata_getnobj("omsobj:SASLibrary?@Name ? ‘’ ", n, uri_lib);  Follow association to get directory path of each rc = metadata_getattr(uri_lib, 'Name', name); rc = metadata_getnasn(uri_lib, 'UsingPackages', 1, uri_path); rc = metadata_getattr(uri_path, 'DirectoryName', path);

14. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Output of directory path values Library name= groc path=Data\DWAD\Grocery Library name= sastest path=Data\sastest Library name= osas2_db path= No Path ! It is a DBMS library definition

15. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Important to query the correct Attribute Base SAS library will have DirectoryName attribute DBMS library will have SchemaName rc = metadata_getattr(uri_path, 'SchemaName', schema);

16. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. A more interesting report  Requirement: Data warehouse ETL maintenance will add new jobs and change existing ones Partial promotion needs to promote all changed jobs and tables; we know which ones have been changed. But other – unchanged – jobs may use tables that have new/altered columns. Code must be regenerated to avoid errors or warnings at run time. How can we know which tables every job uses as input or output? Point-and-click Impact Analysis is tedious for a large number of tables!

17. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Staring point – examine the metadata of a job Input Tables Output Tables

18. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. More complex jobs also have intermediate tables Input to Loader is Output from Sort Object is a WorkTable

19. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. To complete the programming, we need to… Get the URI for each Job in the repository and navigate the Steps and Transformations associations to get every processing step Read all the ClassifierSources and ClassifierTargets Associations for each Transformation to identify the Tables Filter out WorkTable references – they are all internal to the jobs and of no interest Output the results oNote: the code can be found in the Proceedings or downloaded from www.appliedsystem.co.uk

20. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. A much more complex job…

21. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. … reported as part of a whole repository analysis

22. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Conclusion Metadata we build for our ETL jobs is a rich source of information By using SAS DATA step functions any moderately experienced SAS programmer can access this resource Metadata Browser provides an aid to understanding the structure and relationships in metadata – use it as your guide!

23. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Thank You! Questions? The author can be contacted at:

24. Copyright © 2008, SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.