Oracle for Data Warehousing

Slides:



Advertisements
Similar presentations
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any.
Advertisements

Extreme Performance with Oracle Data Warehousing
Exadata Goals Ideal Oracle Database Platform
Supervisor : Prof . Abbdolahzadeh
Cloud Computing: Theirs, Mine and Ours Belinda G. Watkins, VP EIS - Network Computing FedEx Services March 11, 2011.
Oracle Exadata for SAP.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1.
Living with Exadata Presented by: Shaun Dewberry, OS Administrator, RDC Tom de Jongh van Arkel, Database Administrator, RDC Komaran Hansragh, Data Warehouse.
The Power of Platform Solutions to Transform Higher Education
Introduction to DBA.
1. Aim High with Oracle Real World Performance Andrew Holdsworth Director Real World Performance Group Server Technologies.
Essbase Reporting Jim Kubik Senior Sales Consultant.
A Fast Growing Market. Interesting New Players Lyzasoft.
Oracle Data Warehouse Strategic Update Ray Roccaforte.
Meanwhile RAM cost continues to drop Moore’s Law on total CPU processing power holds but in parallel processing… CPU clock rate stalled… Because.
Data Warehousing - 3 ISYS 650. Snowflake Schema one or more dimension tables do not join directly to the fact table but must join through other dimension.
Microsoft SQL Server x 46% 900+ For Hosting Service Providers
Extreme Performance Data Warehousing
Data Warehousing: Defined and Its Applications Pete Johnson April 2002.
Base Content Slide Larry Ellison CEO, Oracle
© Hitachi Data Systems Corporation All rights reserved. 1 1 Det går pænt stærkt! Tony Franck Senior Solution Manager.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1 Preview of Oracle Database 12 c In-Memory Option Thomas Kyte
The Sun Oracle Database Machine Barry Hodges Senior Solution Architect Oracle New Zealand.
Oracle10g for Data Warehousing Jiangang Luo
SANPoint Foundation Suite HA Robert Soderbery Sr. Director, Product Management VERITAS Software Corporation.
Bob Thome, Senior Director of Product Management, Oracle SIMPLIFYING YOUR HIGH AVAILABILITY DATABASE.
1.
Database Systems – Data Warehousing
Andrew Mendelsohn Senior Vice President Database Oracle Corporation.
Data Warehousing at Acxiom Paul Montrose Data Warehousing at Acxiom Paul Montrose.
Business Intelligence Appliance Powerful pay as you grow BI solutions with Engineered Systems.
Faster and Smarter Data Warehouses with Oracle OLAP 11g.
FlashSystem family 2014 © 2014 IBM Corporation IBM® FlashSystem™ V840 Product Overview.
Oracle Advanced Compression – Reduce Storage, Reduce Costs, Increase Performance Session: S Gregg Christman -- Senior Product Manager Vineet Marwah.
1.
SESSION CODE: BIE07-INT Eric Kraemer Senior Program Manager Microsoft Corporation.
1 Reviewing Data Warehouse Basics. Lessons 1.Reviewing Data Warehouse Basics 2.Defining the Business and Logical Models 3.Creating the Dimensional Model.
Achieving Scalability, Performance and Availability on Linux with Oracle 9iR2-RAC Grant McAlister Senior Database Engineer Amazon.com Paper
 2009 Calpont Corporation 1 Calpont Open Source Columnar Storage Engine for Scalable MySQL Data Warehousing April 22, 2009 MySQL User Conference Santa.
Information systems and management in business Chapter 8 Business Intelligence (BI)
7 Strategies for Extracting, Transforming, and Loading.
Infrastructure for Data Warehouses. Basics Of Data Access Data Store Machine Memory Buffer Memory Cache Data Store Buffer Bus Structure.
CERN - IT Department CH-1211 Genève 23 Switzerland t High Availability Databases based on Oracle 10g RAC on Linux WLCG Tier2 Tutorials, CERN,
SAM for SQL Workloads Presenter Name.
Rajesh Bhat Director, PLM Analytics Applications
WHAT EXACTLY IS ORACLE EXALYTICS?. 2 What Exactly Is Exalytics? AGENDA Exalytics At A Glance The Exa Family Do We Need Exalytics? Hardware & Software.
Introduction to Exadata X5 and X6 New Features
Looking Forward while Looking Backwards Essbase and Exalytics Seldon Goodwin Principal Sales Consultant Oracle Business Intelligence.
Peter Idoine Managing Director Oracle New Zealand Limited.
Oracle Exalytics Business Intelligence Machine Eshaanan Gounden – Core Technology Team.
© 2009 Oracle Corporation – Proprietary and Confidential Agenda Reporting Overview Performance Workspace Dashboards Reports Drill thru Smartview Excel.
Supervisor : Prof . Abbdolahzadeh
Workload-Management für komplexe Data Warehousing Umgebungen
Data Platform and Analytics Foundational Training
5/9/2018 7:28 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS.
Defining Data Warehouse Concepts and Terminology
with the Microsoft BI Ecosystem
Couchbase Server is a NoSQL Database with a SQL-Based Query Language
Delivering Business Insight with SQL Server 2005
Defining Data Warehouse Concepts and Terminology
Blazing-Fast Performance:
DeFacto Planning on the Powerful Microsoft Azure Platform Puts the Power of Intelligent and Timely Planning at Any Business Manager’s Fingertips Partner.
XtremeData on the Microsoft Azure Cloud Platform:
Introduction to SAP HANA
Data Warehouse.
Data Warehousing Concepts
Introduction of Week 14 Return assignment 12-1
Analytics, BI & Data Integration
6/17/ :03 AM © 2004 Microsoft Corporation. All rights reserved.
“Kontrax and Partners”
Presentation transcript:

Oracle for Data Warehousing

Packaged Applications Business Intelligence Performance Management Today’s Reality Analytics Packaged Applications Business Intelligence Custom Reporting Performance Management Data Replication Data Migration Data Warehousing Data Federation Data Marts Data Silos Data Hubs Data Access SQL Batch Scripts Custom Java Fragmented data silos, data-marts, and data-centric applications are the norm Inconsistent data, risk from data exceptions; untrustworthy data Inaccessible information, lack of re-use Lack of visibility to how information assets are used and leveraged Poor business insight, incomplete information for decision making Data Warehouse Data Mart SAP, Oracle PeopleSoft, Siebel, Custom Apps Files Excel XML OLTP & ODS Systems OLAP 2 2 2

What’s the Impact on Your Business? Fragmented data Silos Untrustworthy Data Inaccessible Information Limited Scalability Difficult to Manage Higher Costs Increased Risk Poor Decisions Fragmented data Silos Higher Costs Increased Risk Poor Decision Making Fragmented data silos, data-marts, and data-centric applications are the norm Inconsistent data, risk from data exceptions; untrustworthy data Inaccessible information, lack of re-use Lack of visibility to how information assets are used and leveraged Poor business insight, incomplete information for decision making 3 3 3

Moving toward a Clean Architecture APPLICATIONS ANALYTICS / APPS SERVERS ORCHESTRATION MASTER DATA BPEL PM CRM APPS SCM APPS WEBSITES FINANCIALS DISTRIBUTION CUSTOM APPS OPERATIONS ERP APPS MDM APPLICATIONS Master Data EPM BI ... BUSINESS INTELLIGENCE CLUSTERS ETL Master Data Data Warehouse ETL ESB

Data Warehouse Reference Architecture

Oracle’s Integrated Stack Customer Service Ad Hoc Analysis Interactive Dashboards Performance Management Reporting & Publishing Proactive Detection Data Integration & Management Data Warehousing Business Intelligence Foundation PM and BI Applications Data Mining Storage Compression OLAP Predictive PM Applications BI Applications Partitioning Modeling Data Federation Data Quality ETL/ELT Data Services Database and Middle Tier Servers

Oracle: #1 in Data Warehousing Numbers Show Oracle #1 Database (Again). Oracle Confidential – Do Not Distribute Source: IDC, August 2010 – “Worldwide Data Warehouse Platform Software 2009 Vendor Shares”

Oracle Data Warehouse Customers Retail Communications Financial Services Manufacturing Manufacturing CPG/Healthcare Transportn/Energy Leading companies across virtually every industry have chosen Siebel Analytics. Some of the most sophisticated users of analytics technologies—companies like Procter & Gamble and UPS, for example—have standardized on Siebel as their analytics platform. The benefits these companies are achieving with Siebel Analytics are very significant. Honeywell, for example, has improved customer satisfaction by 30% and increased deal close rate. They have 1,000+ users at all levels of the organization. General Motors is using Siebel Analytics to analyze dealer sales, service, and parts data from 21 disparate data sources, providing 1600 users in the field access to complete business insight they need At Cisco, Siebel Analytics provides 500 top execs with near real-time global picture of business performance and prospects At Royal Bank of Canada -- Integrates over 14 terabytes of data from checking, credit card, and mortgage systems, delivering multiple analytic applications to thousands of users enterprise wide; Millions saved through improved collections and reduced defaults

Oracle Exadata Database Machine Extreme Performance Fastest growing new product in Oracle’s history Server & Storage Integrated Hardware & Software Platform Data Warehousing OLTP Consolidation “After carefully testing several data warehouse platforms, we chose the Oracle Database Machine. Oracle Exadata was able to speed up one of our critical processes from days to minutes.” Brian Camp SVP, Infrastructure Services Knowledge Base Marketing

Oracle for Data Warehousing Optimized for strategic warehousing 25 GB/sec IO bandwidth, with up to 50 GB/sec with Flash Optimized for real-world data loading Read consistency with the ability to load at 5TB/hr Optimized for operational warehousing Advanced indexing capabilities running at 1M IOPS Optimized for advanced analytics Integrated OLAP, data mining, spatial and statistics Optimized for large data sets 10x user data compression

Exadata Hardware Architecture Scaleable Grid of industry standard servers for Compute and Storage Eliminates long-standing tradeoff between Scalability, Availability, Cost Database Grid Intelligent Storage Grid 8 Dual-processor x64 database servers OR 2 Eight-processor x64 database servers 14 High-performance low-cost storage servers 100 TB High Speed disk, or 336 TB High Capacity disk 5.3 TB PCI Flash Data mirrored across storage servers InfiniBand Network Redundant 40Gb/s switches Unified server & storage network

Complete Family Of Database Machines For OLTP, Data Warehousing & Consolidated Workloads Oracle Exadata X2-2 Oracle Exadata X2-8 Quarter, Half, Full and Multi-Racks Full and Multi-Racks 12

Exadata is Smart Storage Compute Intensive Processing Database Server Compute and memory intensive data processing executes in database servers Fully-parallelized joins and aggregations Bandwidth Intensive Searches Exadata Storage Server IO-bandwidth intensive database operations executes in storage servers Exadata Smart Scans and Exadata Storage Indexes filter out data that is not relevant to a query Database servers and Exadata storage work in conjunction to execute SQL Exadata cell is smart storage, not a complete database node

Exadata Database Processing in Storage Exadata storage servers implement data intensive processing in storage Row filtering based on “where” predicate Column filtering Join filtering Incremental backup filtering Scans on Hybrid Columnar Compressed data Scans on encrypted data Data Mining model scoring 10x reduction in data sent to DB servers is common No application changes needed Processing is automatic and transparent Even if cell or disk fails during a query

Exadata Flash Extreme Performance Oracle Database Machine has 5 TB of flash storage 4 high-performance flash cards in every Exadata Storage Server Smart Flash Cache caches hot data Not just simple LRU Knows when to avoid caching to avoid flushing cache Allows optimization by application table Oracle is the First Flash Optimized Database 15

Oracle Database Machine: Optimized for large scans 10 TB of user data Requires 10 TB of IO 1 TB with compression 100 GB with partition pruning Subsecond On Database Machine 20 GB with Storage Indexes 5 GB with Smart Scans 2000X less data needs to be processed

What does Extreme Performance mean for your business? Massive data volumes More granular data Daily data instead of weekly Store data instead of account More history 5 years instead of 1 year New data sources Consumer-level data Entirely new analytics Queries that were never possible now run in minutes Near-real-time data loading

Oracle Database Machine Optimized for strategic warehousing 25 GB/sec IO bandwidth, with up to 50 GB/sec with Flash Optimized for real-world data loading Read consistency with the ability to load at 5TB/hr Optimized for operational warehousing Advanced indexing capabilities running at 1M IOPS Optimized for advanced analytics Integrated OLAP, data mining, spatial and statistics Optimized for large data sets 10x user data compression

Fast data loading Access Transform Load Performant data loading and transformation with in-database ETL Direct flat file access with external tables Transformation inside the database with SQL and PL/SQL Bulk and trickle load Up to 5 TB/hr of raw data loading on a Database Machine Access Transform Load

Transform Data Where Data Resides In-database ETL technology Extract Load Transform Insert Data Pump Transportable Tablespaces Partition Exchange Loading Change Data Capture Distributed Queries SQL*Loader External Tables Table Functions Multi-Table Insert MERGE DML error logging

DBFS - Scalable Shared File System Database Machine comes with DBFS shared Linux file system Shared storage for ETL staging, scripts, reports and other application files Files stored as SecureFile LOBs in database tables stored in Exadata Protected like any DB data – mirroring, DataGuard, Flashback, etc. 5 to 7 GB/sec file system I/O throughput Load into database using External Tables Using DBFS, it is possible to implement a shared file system in the Oracle Database Machine. An example use of such a shared file system, would be an ETL Staging Area. In this case, data is copied onto the DBFS in the Exadata Storage Servers using traditional OS utilities (like ftp, scp, rcp) and then ETL load tool can load the data from the DBFS into the database tables using the eternal tables interface. ETL Files in DBFS ETL More File Throughput than High-End NAS Filer 21

Real Time Data Integration with Oracle GoldenGate Real time extracts from transactional systems Non-invasive on sources Continuous streaming load into ODS Schema of target Latency in seconds Source 1 EMP DEPT On-Disk Logs Oracle GoldenGate Source 2 EMP DEPT Oracle GoldenGate On-Disk Logs

Oracle is uniquely capable of concurrent query and updates report Concurrent small data loads and queries Looks like... OLTP Oracle's read consistency Readers never block writers Writers never block readers Queries are always consistent and auditable No deadlocks Introduced in Oracle V4 (1982) Teradata: lock rows for reads and writes to obtain consistency Budget table update Rollback Segment Before Image update accurate report

Oracle Database Machine Optimized for strategic warehousing 25 GB/sec IO bandwidth, with up to 50 GB/sec with Flash Optimized for real-world data loading Read consistency with the ability to load at 5TB/hr Optimized for operational warehousing Advanced indexing capabilities running at 1M IOPS Optimized for advanced analytics Integrated OLAP, data mining, spatial and statistics Optimized for large data sets 10x user data compression

Data Warehouse Reference Architecture Base data warehouse schema Atomic-level data, 3nf design Supports general end-user queries Data feeds to all dependent systems Application-specific performance structures Summary data / materialized views Dimensional view of data Supports specific end-users, tools, and applications

Data Warehouse Reference Architecture IO-bandwidth intensive workloads Random-IO intensive workloads

Oracle delivers complete capabilities for tactical warehouse queries Fast lookups B-Tree indexes Dimensional queries Star query optimizations Bitmap indexes Bitmap join indexes Aggregate management Materialized Views Cube-organized materialized views

Workload Management for DW Setting Up a Workload Management System Define Workloads Filter Exceptions Manage Resources Monitor Workloads Adjust Plans Execute Workloads Monitor Workloads Adjust Workload Plans IORM RAC OEM DBRM Define Workload Plans The RAC piece includes things like: Services Server Pools (Grid Infrastructure) to provide elasticity (add servers to pool to increase memory) Instance Caging (consolidation) © 2010 Oracle Corporation

Workload Management Request Queue Execute Assign Ad-hoc Workload Each request: Executes on a RAC Service Which limits the physical resources Allows scalability across racks Assign Each request assigned to a consumer group: OS or DB Username Application or Module Action within Module Administrative function Ad-hoc Workload Each consumer group has: Resource Allocation (example: 10% of CPU/IO resources) Directives (example: 20 active sessions) Thresholds (example: no jobs longer than 2 min) Reject Downgrade © 2010 Oracle Corporation

Workload Management Request Real-Time ETL Batch ETL Analytic Reports Assign Execute Execute OLTP Requests Ad-hoc Workload Queue Downgrade Reject © 2010 Oracle Corporation

Workload Management Request Real-Time ETL Queue R-T 10% Batch ETL Analytic Reports Analytic Reports 50% Queue Assign OLTP Requests OLTP 5% Reject Downgrade Queue Ad-hoc 25% Ad-hoc Workload Queue © 2010 Oracle Corporation

Oracle Database Machine Optimized for strategic warehousing 25 GB/sec IO bandwidth, with up to 50 GB/sec with Flash Optimized for real-world data loading Read consistency with the ability to load at 5TB/hr Optimized for operational warehousing Advanced indexing capabilities running at 1M IOPS Optimized for advanced analytics Integrated OLAP, data mining, spatial and statistics Optimized for large data sets 10x user data compression

In-database Analytics Bring Algorithms to the Data, Not Data to the Algorithms Analytic computations done in the database Dimensional analysis Statistical analysis Data Mining Scalability Security Backup & Recovery Simplicity OLAP Statistics Data Mining

Deeper insights from Oracle Data Warehouses Oracle Spatial: GIS data available for analysis and displayed via MapViewer

Deeper insights from Oracle Data Warehouses Oracle OLAP: Hierarchically aware rankings, shares, alerts and time series calculations are easily defined in the cube and queried by OBIEE using simple and efficient SQL

Deeper insights from Oracle Data Warehouses Hierarchically aware rankings, shares, alerts and time series calculations are easily defined in the cube and queried by OBIEE and other tools using simple and efficient SQL Oracle Data Mining: Predictions & probabilities are calculated within database and available for reporting using OBIEE

Deeper insights from Oracle Data Warehouses OLAP Data Mining Spatial  In-Database Analytics Deeper insights for business users Pervasive benefits across stack Simple and seamless embedded analytics Extreme performance and scalability

Oracle Database Machine Optimized for strategic warehousing 25 GB/sec IO bandwidth, with up to 50 GB/sec with Flash Optimized for real-world data loading Read consistency with the ability to load at 5TB/hr Optimized for operational warehousing Advanced indexing capabilities running at 1M IOPS Optimized for advanced analytics Integrated OLAP, data mining, spatial and statistics Optimized for large data sets 10x user data compression

Exadata Hybrid Columnar Compression Highest Capacity, Lowest Cost Data is organized and compressed by column Dramatically better compression Speed Optimized Query Mode for Data Warehousing 10X compression typical Runs faster because of Exadata offload! Space Optimized Archival Mode for infrequently accessed data 15X to 50X compression typical Query Faster and Simpler Backup, DR, Caching, Reorg, Clone Benefits Multiply © 2010 Oracle Corporation

More Data Capacity Systems with Equal User Data Exadata V2 10x Compression Systems with Equal User Data All with Largest Disks, Best Compression Netezza TwinFin 2x to 4x Compression EMC VMAX 3x Oracle Compression Exadata has more disk drives per rack, larger disk drives (2TB) and much better compression. This means that Exadata can hold much more User Data than other systems, and costs much less per user Terabyte. Exadata User Data per SATA rack with 10x compression is 500TB. Teradata has fallen very far behind in compression technology which makes them much more costly for large data environments. Teradata 2580 holds 45 TB per cabinet using max sized 1TB drive and 1.3x compression (taken from Teradata specifications). A single Exadata rack matches the user data capacity of the largest size Teradata 2580 (12s cabinet holding 517 TB user data). The flagship Teradata 5600 is hold even less data per rack than the 2580. There is approximately a 20:1 Ratio of user data per rack comparing Exadata to Teradata 5600. Netezza Twinfin 32 TB Uncompressed per rack, 128 TB Compressed (assuming their maximum 4x compression). Teradata 2580 1.3x Compression

Summary

Enterprise Availability Active Data Guard RAC ASM Secure Backup WAN Flashback Online Redefinition GoldenGate Replication Exadata has fully redundant hardware. Redundant servers, redundant storage servers, redundant network. So any component can fail and the system as a whole will keep running. Our measurements to date show that the hardware failure rate is dominated by disk failures. The Oracle database software tolerates failures by continuing to run when various hardware components fail. For example Oracle RAC continues to run after server failures. ASM mirrors data across storage servers so that the failure of a storage server does not cause an outage of the system as a whole. Oracle has unique capabilities for rolling back erroneous changes called flashback. Oracle has unique capabilities for making changes to databases online called online redefinition. All truly highly available systems should have a remote replica. Oracle has the industry’s leading technologies for creating and maintaining remote replica databases. Golden gate provides a powerful symmetric replication capability. Active Data Guard provides an extremely high performance and simple way to create a readable remote replica database. Redundant Hardware Servers, Storage, Network Database Level HA Tolerate failures and changes Real-Time Active Replica © 2010 Oracle Corporation 42

First Secure Database Machine Moves decryption from software to hardware Over 5x faster Near zero overhead for fully encrypted database Queries decrypt data at hundreds of Gigabytes/second © 2010 Oracle Corporation 43

Standardized and Simple to Deploy All Database Machines are the same Delivered Tested and Ready-to-Run Highly Optimized Highly Supportable No unique configuration issues Identical to config used by Oracle Engineering Runs existing OLTP and DW applications Full 30 years of Oracle DB capabilities No Exadata certification required Leverages Oracle ecosystem Skills, knowledge base, people, partners Ready- to-Run Eliminates the complexity of deploying a high performance database system. Database machines are tested in the factory and delivered ready to run. Because all database machines are the same, their characteristics and operations are well known and understood by Oracle field engineers and support. Each customer will not need to diagnose and resolve unique issues that only occur on their configuration. Performance tuning, and stress testing performed at Oracle is done on the exact same configuration that the customer has ensuring better performance and higher quality. Applications do not need to be certified against Exadata. Applications that are certified with Oracle Database 11.2 RAC will run against Exadata. Very few applications need to certify the storage subsystem underneath a database, and Exadata fundamentally is the Oracle Database with a very fast storage subsystem. Deploy in Days, Not Months © 2010 Oracle Corporation

Faster than DW Appliances 50 GB/sec! Flash Query Throughput GB/sec Uncompressed Data Single Rack Faster Throughput from Disk Much Faster with Flash 50 TB of data fits in Flash Using 10x Query Compression Effective Query Throughput on compressed data is even higher Hundreds of GB/sec Disk Teradata 2580 Netezza TwinFin 12 Exadata V2 Why is Oracle Faster DB Processing in Storage Smart Flash Cache Faster Interconnect (40Gb/sec) More Disks Faster Disks (15K RPM) Exadata simultaneously scans from Flash and Disk to maximum query throughput. Exadata allows flash directives at the table or partition level easily implementing ILM Note that the limiting factor to query throughput with compressed flash cache data is generally CPU. The high rate of IO generally moves the bottleneck out of the IO system.

Oracle Exadata Momentum Rapid adoption in all geographies and industries

Oracle Exadata Database Machine: Changes the Game Database intelligence and massively parallel scaling in the storage tier Using state of the art industry standard hardware Complete, Integrated Data Warehouse, OLTP and consolidation solution High availability Enterprise Security Advanced analytics Innovative new technologies: Hybrid Columnar Compression In-memory parallel execution FlashFire hardware + flash-optimized software