ACT Data Access Architecture

Content Goals and Objectives Assumptions and Dependencies Current State Problem statement Data access usecases (High level requirements) Proposed data Access and Reference Architecture Proposed consumption method Solution for Data access usecases based on proposed architecture Road map

Goals & Objectives Provide a blue print for data access Provide most common use cases and apply new architecture to these use cases Provide road map for implementation

Assumptions & Dependencies Transition plan and specific details will follow after approval of the blueprint document Standard set of services and interfaces have been identified API Manager is available to provide run time governance for services Data governance is defined for DW 2.0 Data models and related meta models are defined

EasyQuery QueryLink ODBC/JDBC Current State Enterprise ODS Enterprise DW 1.0 Cognos BI ETL System of record System of reference ETL Dept systems Applications

Problem Statement Proliferation of shadow systems Stale data Tightly coupled applications Direct database access results in loss of control, long test cycle, communication etc Data integrity issues Access control issues Compliance issues Over reliance on single system - reliability Scalability Cost due to inefficiencies and redundant data systems Tightly coupled batch jobs

Proposed Data Access Enterprise ODS Enterprise DW 2.0 Informatica Source of record Source of reference Web service API Mgr / ESB Dashboards, reports, cubes Workflow, enrichment. queuing Events Pub/sub Message Queue Cognos/BI Data Consumption

Departmental Systems BLACKBAUD COEUS UCPATH ISIS IFIS PPS ODS System of Reference ODS System of Reference Cognos Other Reporting Tools Reporting IDS Data Virtualization JBOSS Tomcat Web services WSO2 DSS.Net WSO2 ESB Informatica Integration ActiveMQ Reference Architecture Query studio Data Query

Proposed consumption method SOA 2.0 –RESTful /SOAP web services –Events ETL ODS Cognos Files (Deprecated) Database access (Limited)

Implication Instead of accessing database (outside the business unit) directly, use the standard interface. Use Data warehouse for reports and for historical data Data will be accessible from ODS up to 6 months In order to publish API to Enterprise store, it need to comply with the enterprise interface standards. In order to publish data to ODS, it needs to comply with enterprise data quality and standards. Files and big data dump are not preferred approaches unless client limitation or location dictates them.

Implication NowFuture 1Access COEUS or Mainframe database directly (Outside Business Unit) Create API and access them using API - Who will create them? - Bottleneck? 2Access DW for accessing any dataAccess ODS for the same data (Limit 6 month worth of data) 3Take a data dump from DWUse Cognos to generate report For analytics, use ?? 4Ability to access historical dataUse system of record?

USE CASES

Scenarios TypeOptions OperationRead Write/Read Data SizeAtomic data Bulk data Report Latencynear real time real time x hours- 1 day Client LocationInternal to ACT Mainframe Campus Outside campus Data ClassificationSensitive (HIPPA, PII, FERPA) Public Internal Confidential SourceACT Dept UCOP

Use case #1- Transaction (Bulk data) Source being updated Client LatencySecurity ImplementationConsumption Example MainframeNA Non- Realtime NA Load the data into ODS File/ODS NA RealtimeNA Direct to the target Web service NA Direct to the target Web service

Use case #2- Transaction (Atomic Write or Read-Write-Read) Client Source being updated LatencySecurity Consumption Implementation Example NAMainframe Coeus RealtimeNA Web ServiceAgainst source

Use case #3- Read Only (Bulk data for subsequent analysis ) SourceClient LatencySecurity Implementation Consumption Example ACT Dept ACT & Dept NA 1-n hour/day (non-real time) No sensitive data Source system or Secondary system For historical data => DW For current data => ODS or transactional system File/Database Access

SourceClient LatencySecurity Implementation Consumption Example ACT Dept ACT & Dept NARealtimeNA From the system of source or secondary data source using CDC. Service can be built upon department, enterprise source systems or ODS or more than one data sources. Web service Non- Realtime NACan be implemented against ODS Web service Use case #4- Read Only ( Atomic or limited number of records )

Read Only Use case #5- Reports SourceClient LatencySecurity Implementation Consumption Example ACT Dept ACT & Dept NA Data will be migrated to data warehouse using Informatica. Cognos reports will be generated against DW Reports generated through Cognos

Use case #6- Event Notification SourceClient LatencySecurity Implementation Consumption Example ACT Dept Java Client Web UI RealtimeNA Events can be generated against system of records or against Message Queue

Road map Platform ESB Messaging Cognos ODS DW 2.0 API Manager

Q & A

COMPONENTS

Operational Data Store (ODS) Definition: An ODS is an integrated, subject- oriented, volatile, current-valued structure designed to serve operational users Characteristics: Source of reference with data supplied by source of record Data reflection of transactional systems Data typically fully normalized Minimal latency Data content is for enterprise usage Data defined per enterprise perspective Current data Data quality ensured by upstream validation

ODS Usage: Hub for enterprise wide consumption of operational data Access primarily by services/system accounts Minimal reporting Provides primary source for Data Warehouse Considerations: Supports tactical decisions Integration grows as subject matter introduced Relatively simple to deploy but expect more difficulty as data currency demands grow Provide rich metadata Govern content tightly – avoid data dumping ground

Data Warehouse (DW) 2.0 Definition : A central data store used for Business Intelligence (BI) reporting and analysis. Includes integrated historical and current data from multiple subject areas. DW 2.0 is a new instantiation of the enterprise warehouse. Characteristics: Contains history per retention requirements Framework is dimensional model Optimized for query performance Primary data source for analytics Data supplied by ETL Significant aggregation Data quality ensured Non-volatile

Data Warehouse (DW) 2.0 Usage : Business Intelligence primarily via Cognos Historical analysis Operational reporting Source for predictive analytics Considerations: Supports strategic decisions Role based access Integration grows as subject matter introduced Relatively complex to deploy with additional complexity based on subject areas introduced Fully utilize database capabilities for optimization Retention policies required to determine history Govern content tightly – avoid data dumping ground

Enterprise Service Bus Definition : ESB is a style of integration architecture that allows communication via a common communication bus that consists of a variety of connections between providers and users of services Characteristics: Content-based routing Protocol transformation, adapters Service aggregation Security-encryption, signing, authorization, authentication Reliable delivery of messages Message Exchange Patterns Message enrichment and validation Versioning Throttling

Enterprise Service Bus Usage : Integration Security Queuing and Throttling Business Partner Interaction Access SOAP Services Considerations: Canonical Data model Application Adapters Number of applications being integrated Stateless services, state attached to the message

API Manager Definition : API management is the process of publishing, promoting and overseeing application programming interfaces (APIs) in a secure, scalable environment. It also includes the creation of end user support resources that define and document the API Characteristics: Create a Store of all Available APIs Route API Traffic in secured fashion Govern Complete API Lifecycle Supports API Subscription workflow Monitor API Usage and Performance Support for Creating multi-tenanted APIs Caching and Throttling

API Manager Usage : Application, application developers and application users outside IT governance Self service security Considerations: API versioning strategy Read vs Write APIs API caching Token expiration API publishing and subscription approval work flow Governance