Building a Corporate Metadata Repository Past Present Future CMR Sam Highsmith Principal Researcher Statistical Research Division U.S. Census Bureau

Agenda The Past The Present Requirements for 2000 Questions

What is the Corporate Metadata Repository? Electronic card catalog Developed in conjunction with work from Sweden, Canada, Australia, and UN/ECE Metadata Workshop Implements Business Data Model for Surveys and Censuses Provides a process for registering data set, variables, and documents for access and reuse

Past Milestones Built a Metadata Repository model Built a Web enabled Prototype Formal Agreements with other Bureau users DADS adopted the Metadata Repository model Developed a Transition to Production plan Contract with Oracle to build a CMR Economic Directorate Pilot Application

Metadata Repository Model Business Data Model –CASRO, SRD, consultant, Reinvention Lab –Survey and Census business model Data Element Registry –ISO Built BDM and DER, then merged to CMR Modeled with ERD tool

Milestones PROTOTYPE Interface to Multiple Statistical Information Systems - DADS, FERRET, PCDOCS Table of Contents organization Search capabilities Oracle generated database for repository Completely Web based

Milestones Formal Memorandum of Understanding with major Bureau directorates

Milestones DADS adopted the Metadata Repository model Implemented by Oracle Consulting for AFF –Dissemination tool for 2000 Census data –Accessible from DADS American Fact Finder is completely metadata driven !!!!!!!!!!!!

Milestones Developed a Transition to Production plan Metadata support staff metadata providers in directorates Still waiting…...

Milestones Built a Metadata Repository model Built a Web enabled Prototype Formal Agreements with other Bureau users DADS adopted the Metadata Repository model Developed a Transition to Production plan Contracted with Oracle to Build CMR

Review of CMR Implementation Strategy Current Business Process does not include an Integrated Metadata Business Process internally developed systems customized commercial systems CASES variety of programming languages individual tool of choice internally developed systems customized commercial systems CASES variety of programming languages individual tool of choice CATI CAPI Mail PAPI OCR ICR CADE CSAQ TDE PFIRS CATI CAPI Mail PAPI OCR ICR CADE CSAQ TDE PFIRS internally developed systems SAS DEVSURV COBOL FORTRAN DECForms StEPS ECON DW individual tool of choice internally developed systems SAS DEVSURV COBOL FORTRAN DECForms StEPS ECON DW individual tool of choice AFF CENSAS FERRET Econ DW CD-ROM Internet AFF CENSAS FERRET Econ DW CD-ROM Internet Census 2000 ACS Demographic Surveys Econ Census Econ Surveys Design Collect Process Share

Review of CMR Implementation Strategy Benefits of a CMR A C T I V E Corporate Metadata Repository (CMR) Statistical MD Geographic MD Technical MD Business Rules Product Layouts A C T I V E Corporate Metadata Repository (CMR) Statistical MD Geographic MD Technical MD Business Rules Product Layouts Launch Dissemination Systems Support One View to Internet Users Support “Virtual Statistical Agency” and sharing data/MD across web sites Support Generation of All Kinds of Products, including Reimbursables Support sophisticated analysis of BOC data (e.g. time series and cross dataset comparisons) “Faster to Market” through reuse

Proposed Technical/Software Architecture Four Ways an Application Can Use CMR Metadata Tightly Coupled with CMR Loosely Coupled with CMR 1. Application written against CMR - uses it directly for metadata access and maintenance. 2. Application uses same extensible CMR physical model - can replicate metadata from CMR. (e.g. DADS/AFF,ECON,LEHD) 3. Application communicates with CMR through an API to exchange metadata. (e.g. FERRET) 4. Application communicates with CMR using a standard XML-based metadata interchange. (future - e.g. FEDSTATS,BLS...)

CMR Meta-Model Meets ISO/IEC Data Element Registry standards Meets FGDC Content Standard for Digital Geospatial Metadata (CSDGM) Can support emerging metadata standards Extensible Shareable COTS where possible

Economic Directorate Pilot Pilot versus Prototype Focus on two stages of the information collection life cycle of the ASM Added functionality to two existing METADATA applications –Dissemination and Refics Used JAD, RAD, code generation The “workers” jumped on board! Found multiple existing metadata applications

A METADATA EXAMPLE FROM The Annual Survey Of Manufactures

Metadata Reuse

Computerized Self Administered Questionnaire Metadata Reuse

Metadata Reuse

ASM IRACS Metadata Reuse

Metadata Reuse

A FewViews of the Pilot Application

Hint: A production system! Year 2000 Planning What are we building this year? Hint: A production system!

Metadata Portal Site Web-enabled Access to all CMR tools Promote knowledge management Secure access (role-based & by individual) Support Collaboration and Personalization: BOC Public Access to Tools and Folders Collaboration areas for BOC Projects and Organizations Personalization –Customize own CMR Web Site to short-cut access and add personal work –Publish personal work, if desired

Metadata Management DEPARTMENT EMPLOYEESSALARY DEPT EMPSAL Meta-ModelManagement RepositoryManagement ArchitecturalSolutionsManagement Admin Tools Bulk Load & Unload Services Browsing Tools Metadata Exchange Services u u Publish on Web u u Search structured & unstructured MD u u Drill-through to actual data u u Launch Applications Security u u Network u u System u u Repository u u Metadata u u Fine Grain Ownership u u Object-level Versioning u u Multi-path Versioning u u Object-level Check-in/Check-out Configuration Management V1.0 V1.1V2.0 u u Synchronize Repositories u u Master-Slave Relationships u u Same Meta-Model u u Meta-Model Bridges u u XML-based Exchange u u Formatted/Delimite d ASCII File u u XML-based File u u Publish in many different standard formats u u Create, Read, Update, Delete u u Archive u u Backup/Recovery

Object Layer and Open API Object layer over the CMR physical repository tables –The underlying meta-model could change with minimal impact to the CMR tools and interfaces accessing the object layer. Provide fine-grain, role-based metadata security for access by CMR tools Publish an object oriented Open API for developers Use wizards and generation tools as much as possible Use meta-model generation tools (preferrable)

Metadata Management Web-enabled Meta-model generated application COTS repository administration Supports CRUD manipulation of metadata Supports copying and versioning metadata objects Supports storage and retrieval of unstructured metadata Implements business rules, audit trails, and metadata status for simple work flow The CMR Metadata Management tool will enable the BOC business metadata owners and stewards to maintain their own metadata.

Oracle Designer Generated HTML/JS Applications Logical Models Middle Tier Deployment Server Tier Deployment Physical Models Client Tier Deployment Functional Requirements Use Cases UML Object Model Server Model Web Modules CMR Repository TAPI (PL/SQL) PL/SQL generating HTML & JS Application Code OAS Environment w/ PL/SQL Cartridge & HTTP Listeners Web Browser HTML Application HTTP Net8 View Layer Created/Generated using Oracle Designer Hand coded Created/Generated using Rational Rose

Metadata Browsing and Reporting Web-enabled Meta-model generated application Supports textual and structured metadata searches Supports nested searches Supports hierarchical, tree-based browsing (drill up/down) Supports search and browsing of Tier 1 documents (accessible in the CMR repository or on the Census Intranet/Internet) The CMR Metadata Browsing and Reporting tools will provide read-only metadata searching, hierarchical browsing, and advanced analysis and reporting of the CMR metadata.

Metadata Open Interchange Use an open standard technology (such as XML) to interchange metadata with external systems Support bulk metadata object level refreshes Support metadata validation and correction in staging area prior to final load Enforce role-based, object level security Support web-enabled log of input/output activity Web-enable UI as much as possible The CMR Metadata Open Interchange provides an open, standards-based method for interchanging metadata between the CMR and external systems.

CMR base SAS Web-enabled UI Output metadata for a statistical dataset for use by base SAS Input metadata for a statistical dataset from base SAS Support metadata validation and correction in staging area prior to final load Enforce role-based, object level security Support web-enabled log of input/output activity CMR and SAS metadata interchange tool will enable the metadata owner to perform a bi-directional metadata exchange, and will establish a common mechanism of loading metadata into SAS system and pushing it back to CMR repository.

Target: Production in 2000 ? Questions ?