OBIEE Architecture

Prerequisites* What is Business Intelligence? What are BI needs and challenges? What is Data Warehousing? What are the traditional ways to organize Data? What is the difference between Transactional and Analytical Systems? What is a Dimension / Fact / Star Schema? What are various available BI tools in the industry? * Should have covered by E0 and E1 levels

OBIEE - Unified Business Intelligence Infrastructure Oracle BI Server Proactive Detection and Alerts Disconnected Analytics Ad-hoc Analysis MS Office Plug-in Reporting & Publishing Interactive Dashboards OLTP & ODS Systems Data Warehouse Data Mart SAP, Oracle PeopleSoft, Siebel, Custom Apps Files Excel XML Business Process Multidimensional Calculation and Integration Engine Intelligent Caching Services Simplified Business Model and Abstraction Layer Intelligent Request Generation and Optimized Data Access Services Web Services Multi-Dimensional Sources Advantages: Consolidate and standardize BI tools Seamless user experience across products Unified metadata and shared services - 3 -

Oracle BI EE Architecture Web Browser Javascript for Usability & Interactivity Web Server (IIS, Tomcat, Websphere, iPlanet) SAW Bridge (J2EE/ISAPI) XML, HTML, XLS, PDF, TXT over HTTP/HTTPS Oracle BI Web TCP/IP (SSL) SOAP Web Services, XML and URL Interface Web Catalog Service External Applications and Portals Oracle Interactive Dashboards Oracle Answers HTML, SOAP over HTTP/HTTPS Oracle Delivers Server XML Framework Scheduling/Event Services User Profiling, Security and Session Mngmt Agent Execution Logic TCP/IP (SSL) Cache Services (Web) & Connection Mngmt Device Adaptive Content ODBC over TCP/IP (SSL) Oracle BI Server Oracle BI Administration Oracle BI Cluster Controller Logical SQL ODBC/JDBC (Logical Business Model) Security Services Query Govern. Load Balancer Metadata Interchange Session Management Metadata Management Services Intelligent Request Generation Multi-User Development Services Cache Services Logical Request Generation Metadata Documentation Services Navigator Server Management Services This is a detailed view of the BI EE architecture. You can see the various data sources at the bottom, you have the BI server as the middle tier, and the BI Web Server which drives the end user experience at the top tier. Also, there is the Delivers Server, which is the engine that drives proactive detection, intelligent workflow, and alerting capabilities in Oracle Delivers, and the BI Server administrator, where the BI Server and metadata is managed. Multi-Pass / Sub-Request Logic System / Perf Monitoring Fragmentation Optimization Externalized Authentication Aggregate Navigator LDAP DB Authentication Optimized Query Rewrites Custom Authenticator Execution Engine Data Source Adapters ODBC, CLI, OCI, XML, MDX Analytical and Operational Data Sources

Oracle BI EE Architecture continued… Web Browser Javascript for Usability & Interactivity Web Server (IIS, Tomcat, Websphere, iPlanet) SAW Bridge (J2EE/ISAPI) XML, HTML, XLS, PDF, TXT over HTTP/HTTPS Oracle BI Web TCP/IP (SSL) SOAP Web Services, XML and URL Interface Web Catalog Service External Applications and Portals Oracle Interactive Dashboards Oracle Answers HTML, SOAP over HTTP/HTTPS Oracle Delivers Server XML Framework Scheduling/Event Services User Profiling, Security and Session Mngmt Agent Execution Logic TCP/IP (SSL) Cache Services (Web) & Connection Mngmt Device Adaptive Content ODBC over TCP/IP (SSL) Oracle BI Server Oracle BI Administration Oracle BI Cluster Controller Logical SQL ODBC/JDBC (Logical Business Model) Security Services Query Government Load Balancer Metadata Interchange Session Management Metadata Management Services Intelligent Request Generation Multi-User Development Services Cache Services Logical Request Generation Metadata Documentation Services Navigator Server Management Services Since we’ve talked so much about the enterprise information model, before I dive into how the BI server works, I want to discuss what this metadata looks like and how it is administered. Multi-Pass / Sub-Request Logic System / Perf Monitoring Fragmentation Optimization Externalized Authentication Aggregate Navigator LDAP DB Authentication Optimized Query Rewrites Custom Authenticator Execution Engine Data Source Adapters ODBC, CLI, OCI, XML, MDX Analytical and Operational Data Sources

Oracle BI EE Architecture Web Browser Javascript for Usability & Interactivity Web Server (IIS, Tomcat, Websphere, iPlanet) SAW Bridge (J2EE/ISAPI) XML, HTML, XLS, PDF, TXT over HTTP/HTTPS Oracle BI Web TCP/IP (SSL) SOAP Web Services, XML and URL Interface Web Catalog Service External Applications and Portals Oracle Interactive Dashboards Oracle Answers HTML, SOAP over HTTP/HTTPS Oracle Delivers Server XML Framework Scheduling/Event Services User Profiling, Security and Session Mngmt Agent Execution Logic TCP/IP (SSL) Cache Services (Web) & Connection Mngmt Device Adaptive Content ODBC over TCP/IP (SSL) Oracle BI Server Oracle BI Administration Oracle BI Cluster Controller Logical SQL ODBC/JDBC (Logical Business Model) Security Services Query Govern. Load Balancer Metadata Interchange Session Management Metadata Management Services Intelligent Request Generation Multi-User Development Services Cache Services Logical Request Generation Metadata Documentation Services Navigator Server Management Services Next I’d like to zoom in and discuss in detail the BI Server, which is the heart of the BI EE platform. The BI Server performs several important functions, but boiling it down to the simplest terms what the BI Server does it process inbound “logical” SQL requests for information, build and execute optimized queries for one or many physical data sources, and return the result set to the end users. To an end user tool, the BI Server presents itself like a single database, with “subject areas” of information available as single tables that can be queried through a SQL-based API. This simplified view of information could be coming from data that is quite complex, stored in different physical structures including relational, multi-dimensional, flat files, etc. Let’s walk through the various functional components of the BI Server to get an understanding of how the platform works. Multi-Pass / Sub-Request Logic System / Perf Monitoring Fragmentation Optimization Externalized Authentication Aggregate Navigator LDAP DB Authentication Optimized Query Rewrites Custom Authenticator Execution Engine Data Source Adapters ODBC, CLI, OCI, XML, MDX Analytical and Operational Data Sources

Simplified Logical Model Oracle BI Server Simplified Logical Model The Oracle BI server presents as a logically modeled database enabling the Common Enterprise Information Model. This provides powerful calculation, computation and federation capabilities to 3rd party tools as well as our own. One of the most important and valuable services the BI EE Server provides is a single, enterprise information model that we previously mentioned. This model provides several benefits: It provides a simplified “business view” of information available to all BI tools and all users in the organization regardless of physical data locations or structures. In this way, you can almost think of the business model as a sort of firewall between the users of the information and all the mechanics of the physical data access layer. Data is separate from the model. So let’s say that you have data on inventory in a certain operational data store that is mapped in, and then someone decides that system is going to be retired, so you have to map to a different source to get that inventory data. You can simply remap to the new schema, and it won’t impact all the end user reports because there is a separation or abstraction layer between the physical data layer and the logical business model layer. This allows a high degree of flexibility for IT to make changes to data structure without breaking thousands of end user reports, dashboards, etc. It provides a place where business metrics and calculations are defined, enabling more consistency and alignment in the organization. For example, let’s say the organization has a particular way in which they define customer value, or product profitability. These calculations and metrics can be defined in the metadata and made available to users – define once, deploy everywhere. The enterprise information model presents itself as a single logically modeled database of information. Complex calculations and time series analysis like “% Change vs. Year Ago Revenue” does not need to be calculated by end users, but simply selected for analysis just like any other column (dimension or measure) in the model.

Oracle BI EE Architecture Web Server (IIS, Tomcat, Websphere, iPlanet) Oracle BI Server Intelligent Request Generation Externalized Authentication Data Source Adapters Navigator Logical Request Generation Aggregate Navigator Fragmentation Optimization Multi-Pass / Sub-Request Logic Optimized Query Rewrites Session Management Metadata Interchange Logical SQL ODBC/JDBC (Logical Business Model) Cache Services LDAP DB Authentication Custom Authenticator Oracle BI Administration Analytical and Operational Data Sources Metadata Documentation Services Metadata Management Services Multi-User Development Services Server Management Services ODBC over TCP/IP (SSL) System / Perf Monitoring Oracle BI Cluster Controller ODBC, CLI, OCI, XML, MDX Oracle BI Web SOAP Web Services, XML and URL Interface Cache Services (Web) & Connection Mngmt XML Framework Web Catalog Service User Profiling, Security and Session Mngmt Oracle Interactive Dashboards Oracle Answers Load Balancer Security Services Query Govern. Oracle Delivers Server Scheduling/Event Services Agent Execution Logic Device Adaptive Content TCP/IP (SSL) Javascript for Usability & Interactivity Web Browser XML, HTML, XLS, PDF, TXT over HTTP/HTTPS External Applications and Portals HTML, SOAP over HTTP/HTTPS Execution Engine SAW Bridge (J2EE/ISAPI) TCP/IP (SSL) Intelligent Request Generation Integration of Real time and warehouse Data Integration into Enterprise Portals Reads in system catalog Multiple sources Optimized SQL generation Regardless of Schema Function to use appropriate data sources/Compensation DB2 Supply Chain DM Teradata OLAP Oracle ERP. XML Data Source Excel Spread- Sheets SQL Server Acxiom Siebel Operational

Oracle BI Server Oracle BI Server Siebel Analytics Server Multi-source tuning and load balancing Parallel Execution Engine Intelligent multi-level Caching High performance Siebel Server execution Scheduling Services (Batch and/or Event-Driven) Seamless aggregate and fragment navigation Oracle BI Server Common Enterprise Information Model Metadata Services Multidimensional Calculation Engine Aggregation and Integration Services Intelligent Request Generation and Optimized Data Access Services Clustering Services Parallel Query Execution Engine Intelligent Multi-Level Caching Services Data Mining Services Simplified business model view and Open Intelligent Interface Business calculation & analytic engine Without managed data store Revolutionary request generation and optimized data source access Insight from largest data warehouses and across data sources, systems Analytical, Operational, Transaction, External Relational, MDX, Host, XML, Other Rapid implementation/customization and extensible architecture Market-leading scalability, availability, reliability, performance and TCO

Oracle BI Server Common Metadata Oracle BI Server Physical to logical business model “Roadmap to data” Historical data Real-time data Inventory data Siebel Analytics Server Multi-source tuning and load balancing Parallel Execution Engine Intelligent multi-level Caching High performance Siebel Server execution Scheduling Services (Batch and/or Event-Driven) Seamless aggregate and fragment navigation Oracle BI Server Common Enterprise Information Model Metadata Services Multidimensional Calculation Engine Aggregation and Integration Services Intelligent Request Generation and Optimized Data Access Services Clustering Services Parallel Query Execution Engine Intelligent Multi-Level Caching Services Data Mining Services

Oracle BI Server Oracle BI Server Siebel Analytics Server Multi-source tuning and load balancing Parallel Execution Engine Intelligent multi-level Caching High performance Siebel Server execution Scheduling Services (Batch and/or Event-Driven) Seamless aggregate and fragment navigation Oracle BI Server Common Enterprise Information Model Metadata Services Multidimensional Calculation Engine Aggregation and Integration Services Intelligent Request Generation and Optimized Data Access Services Clustering Services Parallel Query Execution Engine Intelligent Multi-Level Caching Services Data Mining Services Calc & Integration Engine Function ship to appropriate data sources/Compensation Optimized performance SQL < SQL + Oracle BI Server Time Series, Level based Measures

Oracle BI Server Oracle BI Server Siebel Analytics Server Multi-source tuning and load balancing Parallel Execution Engine Intelligent multi-level Caching High performance Siebel Server execution Scheduling Services (Batch and/or Event-Driven) Seamless aggregate and fragment navigation Oracle BI Server Common Enterprise Information Model Metadata Services Multidimensional Calculation Engine Aggregation and Integration Services Intelligent Request Generation and Optimized Data Access Services Clustering Services Parallel Query Execution Engine Intelligent Multi-Level Caching Services Data Mining Services Intelligent Request Generation Optimized SQL generation e.g., “Show me sales through today for my highest value customer. Assure sufficient inventory.” Historical sales Today’s sales Rel. to available inventory DM/ DW ODS Inv. System

Sample Request Processing User views a dashboard or submits an Answers request The OBI Presentation Server makes a request to the OBI Server to retrieve the requested data The OBI Server using the .rpd file, optimizes functions to request the data from the data sources The OBI Server receives the data from the data sources and processes as necessary The OBI Server passes the data to the OBI Presentation Server The OBI Presentation Server formats the data and sends it to the client Clients 1 OBI Presentation Server 2 OBI Server 3 Data Sources 6 5 4

A Day in the Life of a Query Next, let’s discuss the Intelligent Request Generation function and examine the various steps that go into processing an inbound logical SQL request. The first step in the process is the generation of the logical SQL query itself. Logical queries are defined in the Oracle Answers product, shown on the right here. Users select columns from a simple business model. These queries, once defined, are saved and can be embedded in other BI EE end user products such as in Interactive Dashboards. A logical SQL query could also come from another BI tool, SQL query tool, or from Excel.

A Day in the Life of a Query A logical request comes in asking for Brand, Closed Revenue (GL) Service Requests (Call Center) Share of Revenue (Derivation) In this case we’re looking at a query based on Brand, Closed Revenue from the General Ledger, Service Requests from the CRM system, and Revenue Share, which is a multi-pass query. The first thing that happens is the request comes into this Logical Request Generation engine, and it determines how to process the request

A Day in the Life of a Query Oracle BI Server Cache The Server will check to see if this request can be fulfilled via a prior request using Server Cache. This is an on-demand intelligent shareable and secure cache that can support: Subsets, Aggregations and Derivations The first thing the server does is check cache. The cache can satisfy requests that are aggregates of other requests, subsets of other requests, or derivations, and it is secure and personalized for every user. So it again acts as sort of a firewall to the database where you are not needing to hit that iron every time a user has a request. And the nice thing is that as more users get onto the system, the system gets faster. The cache is a runtime concept verses a premeditated concept, so this does not take away from the importance of good data modeling and good aggregates that are in place in the database. So as users are on the system, it gets faster. You can also pre-meditatively populate, or “seed,” the cache. So for example, you might say, OK, these are the long running queries or dashboards, so I want to use the Delivers scheduling capability to run queries periodically to “seed’ the cache with the answers to those longer runnig queries. You might also want to seed the cache each morning so that when workers start their day the first users have the same performance as uses will have once the system has been at load for a period of time. Additional background / questions: When is cache purged? Answer: Cache can be purged on either a time basis, so it could be every 2 hours or every 2 minutes, or every 5 seconds – or it could be on an event basis, so every time the data warehouse is updated cache is purged for example. In addition, this can be done by data element, so some data could be purged every 2 seconds and some could be purged once per day.

A Day in the Life of a Query Navigator If it is the request is cannot be fulfilled by cache then the server will intelligently process the request The next thing is the navigator. Once the BI server figures out that the request can’t be satisfied through cache, the BI server now starts to figure out what is the most optimal way to satisfy this request.

A Day in the Life of a Query Multi-Pass/Sub-Request Logic Does this analysis require sophisticated analysis such as an OLAP calculation? [Revenue Share] The Server will determine the best possible way to provide this calculation/computation The first thing it looks at is what is the complexity of this analysis? Is this a multi-pass analysis? Is there a sub-request involved? In this case we’re talking about Revenue Share, so it decomposes that logical request to figure out what is difficult and what is easy from a logical server perspective.

A Day in the Life of a Query Fragmentation Optimization Does this request span across partitions or silos of data? [Closed Revenue and Service Requests] The server will determine based on the request if it should navigate to both or only one. It will automatically resolve fan traps. The second thing is fragmentation optimization. In this case, closed revenue is in one data silo, and service requests are in another data silo. Fragmentation optimization figures out the best way to get this data. Optional Additional Background: Fragmentation Optimization – The BI Server Fragmentation Optimization Engine is also a piece of middleware that sits between the requesting client and the database. The Fragmentation Optimization Engine intercepts the client's SQL and, wherever possible, transforms base-level SQL into fragment-aware SQL. The Fragmentation Optimization Engine understands how to transform base-level SQL into fragment-aware SQL, because the engine uses special business rules within the BI metadata that describes how the database is partitioned across large common fact tables. The fragment-aware SQL can dynamically determine the most efficient table or tables to fetch the data from. Typically, large fact tables (billion rows) are fragmented into smaller more efficient tables partitioned by time (Q1-2002-Frag, Q2-2002-Frag, Q3-2002-Frag, etc.).

A Day in the Life of a Query Aggregate Navigator Is there an aggregate that can better satisfy this request versus hitting the detail? {Brand Summary Table} The Server will apply approprate summary redirection to provide the quickest response. Next, the system determines if the request can be satisfied through an aggregate of some kind. So rather than going down to the one million customer SKU rows of detailed data, the system checks to see if there an aggregate, or an analytical workspace, or an XMLA cube that can satisfy this request? Let’s navigate there as opposed to going down to the detail data. In this example you are looking for Brand summaries, so perhaps that could be satisfied by looking at regional aggregates and adding those up by Brand. So to summarize the Navigator, what the Navigator does is handle this whole decision tree. How complex is the request? What data sources need to be put into play? What is the best way to satisfy this request and are there aggregates that can be used? Then it generates a query plan based on what it believes is the best plan. Optional Additional Background: Aggregate Navigator – The BI Server Aggregate Navigator is a piece of middleware that sits between the requesting client and the database. The Aggregate Navigator intercepts the client's SQL and, wherever possible, transforms base-level SQL into aggregate-aware SQL. The Aggregate Navigator understands how to transform base-level SQL into aggregate-aware SQL, because the navigator uses special business rules within the metadata that describes the databases aggregate portfolio. The aggregate-aware SQL can dynamically determine the most efficient table to fetch the data from (i.e. the table with the least number of records to successfully fulfill a request).

A Day in the Life of a Query Optimized Query Rewrites What can be function-shipped down to the database versus performed /supplemented by the computation capabilities of the Server. The server will provide a consistent set of functionality to the user regardless of underlying datasource. The server then takes this query plan and does re-writes to optimize for the underlying databases. It figures out what can be function shipped down to the underlying databases? Does the database support a RANK or does it not support a RANK? Does it use sophisticated analytical functions or not? Is it an XML source, a text file, or is it in Oracle? And then it will parallelize that and ship that down to the underlying sources as appropriate, using native connectivity.

A Day in the Life of a Query Cache Services The request once executed will now be available to other users as a cache entry. Last but not least, the server takes that result and puts it back in the cache for the next user. So that a day in the life of a query.

Oracle BI Server System Services Clustering: Add stacks as needed in a share nothing clustered environment. Session Management and Governance: Query throttling via connection pools and authorization Security Services: Integration with OID, LDAP with sophisticated data driven personalization mechanisms. In designing the BI platform, there’s a recognition that it needs to be a first class citizen in terms of the corporate network. There is native load balancing within the BI server. You’ve got session management, as well as query governance so different types of users can be restricted in terms of the queries they can send. It’s got to be able to fail over gracefully, and distribute load as appropriate. Likewise, companies have made investments in other products and so the idea is to leverage those investments. You can leverage all of your existing security models, whether that be for personalization from an authorization perspective, or for authentication via LDAP or ADSI.

High Availability Deployment Scheduler Database Web Catalog Replicated Repository Application DW Oracle Analytics Web Oracle BI Server Scheduler Process Stack 1 Web Server Load Balancer Cluster Controller Other Mart/DW Oracle Analytics Web Oracle BI Server Web Server Stack 2 OAW Replication Agent Web Catalog Master Repository Single Sign-on Server Users Here’s an example of a high availability deployment. The goal here is to do share nothing clustering, so instead of having to take the entire stack and move it on a larger box, what you are able to do is add capability on the fly. So for example, if we’re now at 500 users and we need to go to 1000 to double our capacity, you can just add another stack. That’s the goal of share nothing. Not only that, we can front end the web server with a load balancer. Here you also have an example of single sign on, and we have a variety of mechanisms that keep things in sync. Background / additional information: What is the replication agent? Answer: In a share nothing architecture you need to have replication across stacks. There are two metadata files, one is the web catalog which contains end user generated files (reports, dashboards, alerts, etc) and one is the metadata repository (which contains the BI server metadata, or common enterprise information model as we referred to it earlier). There are two agents that run to keep these synced - for the repository metatdata it’s the cluster controller and for the web catalog it’s the replication agent that maintains that integrity. Administration Tool LDAP Server

OBIEE Clustering OBIEE Clustering Components Cluster Controller Primary Cluster Controller Secondary Cluster Controller Clustered BI Servers Master BI Server BI Scheduler Cluster Manager Failover Mechanisms Cluster Aware Cache -OBIEE Cluster Server Components: Cluster Controller Primary Cluster Controller Secondary Cluster Controller Clustered BI Servers(upto 16 bi servers) Master BI Server(to open online rpd) BI Scheduler(active Passive mode) Cluseter Manager(thru admin tool) Note: All components in Same LAN -Architecture - Failover Mechanisms BI Presentation Services Failure Web Clients iBots BI Server Failure BI Presentation SErvices Administration Tool 3rd Party Clients Master BI Server Failure BI Scheduler Failure -Cluster Aware Cache

Webcat Replication Scenario: Instance 1 – Master Instance Sharing Application Instance 2, Instance 3 – Secondary Clustered Instances exposed to Users ‘Sawrepaj’ setup for the replication on both Windows and Unix environments for performing replication as below /shared: Instance1 to Instance2 /shared: Instance1 to Instance3 /users: Instance2 to Instance3 /users: Instance3 to Instance2 Edit instanceconfig.xml to enable Replication Maintain Log Files of Replication