Next Generation Oracle platform for American Airlines’ Siebel Systems (CAS1151) Reza Abouk, Sr. Database Engineer, American Airlines Amjad Khan, Master Principal Cloud Platform Specialist, Oracle Confidential – Oracle Internal/Restricted/Highly Restricted

Program Agenda 1 Siebel Systems & Current Infrastructure Cloud Objectives & Scope POC Environment, Setup and Steps POC Results & Findings Q & A 2 3 4 5 Confidential – Oracle Internal/Restricted/Highly Restricted

Program Agenda 1 Siebel Systems & Current Infrastructure Cloud Objectives & Scope POC Environment, Setup and Steps POC Results & Findings Q & A 2 Good afternoon. My name is Reza Abouk, and I am a Sr. database system engineer within Customer Technology Infrastructure group at American Airlines. Today I like talk about the This presentation will be mostly focused on the infrastructure for two Siebel systems at American Airlines One AAdvantage Loyalty program which supports over 100 million AAdvantage members, making it the largest Siebel Loyalty implementation in the world. And the other Siebel system is American Airlines Admirals club members and guests 3 4 5 Confidential – Oracle Internal/Restricted/Highly Restricted

Siebel systems AAdvantage Loyalty program Loyalty program supports over 100 million AAdvantage members, making it the largest Siebel Loyalty implementation in the world. American Airlines Sales & CRM Admirals Club Travel Agencies & Corporates Customer Relations

American Airlines Sales CRM Siebel System Current State Architecture - SPARC Super Clusters Primary DR / Stage Primary – Data Center SPARC Super Cluster T5-8 Half Rack (2 nodes) 128 SPARC T5 cores (each node 4 x 16 cores) 2 TB RAM 4 Exadata Storage Servers (144TB HC or 28.8TB HP) 80 TB ZFS Storage Appliance 3 x InfiniBand 36 port switches 12.8 TB Flash Secondary – Data Center (DR & Stage) SPARC Super Cluster T5-8 Half Rack 128 SPARC T5 cores

Siebel SSC Application Layout Current State Architecture - SPARC Super Clusters PROD / Stage DR Half Rack STAGE/ Prod DR Half Rack Node 1 Node 2 Node 1 Node 2 WEB 4 cores GTW 4 cores 4 WEB 4 cores GTW 4 cores WEB GTW WEB GTW APP 1 APP 2 APP 1 APP 2 APP 1 4 cores APP 2 4 cores APP 1 4 cores APP 2 4 cores Prod Each half rack T5-8 has four logical domains (ldoms), first two are used for Web and App layer The last two domains are used for a two node RAC database cluster APP domains include Two gateways and 8 Application servers WEB GTW WEB GTW WEB GTW WEB GTW Prod DR APP 1 APP 2 APP 1 APP 2 APP 1 APP 2 APP 1 APP @ Prod DB 16 cores 128 vcpu Prod DB 16 cores 128 vcpu Prod DR DB Prod DR DB Stage Prod DB Stage DR Stage DR DB 16 cores Stage DR DB 16 cores Stage DB Stage DB 64 core / 512 vcpu 1TB RAM 64 core / 512 vcpu 1TB RAM 64 core / 512 vcpu 1TB RAM 32 core / 512 vcpu 1TB RAM

Program Agenda 1 Siebel Systems & Current Infrastructure Cloud Objectives & Scope POC Environment, Setup and Steps POC Results & Findings Q & A 2 3 4 5 Confidential – Oracle Internal/Restricted/Highly Restricted

November 10, 2018 POC Objective & Scope The objective of this POC is to validate if Oracle Cloud Machine with Exadata Cloud Machine provides same or better throughput/performance that is provided by the current supercluster infrastructure. The above objective will be validated by performing the following on the Siebel applications. Portability/Compatibility for transition from source to cloud platform. Latency test Security (Encryption enabled by default on SqlNet and database TDE) Load test with current data volume. Load test with 100% growth simulated. Application monitoring

Proof of Concept Steps Phase I – Evaluate Siebel Software transition from Solaris to Oracle Linux 6.x Phase II – Siebel POC Performance Testing Provision Oracle VMs (Based on predefined Shape: OCPU/Memory, OS, Storage) from Oracle Cloud @ Customer (OCC) Provision Siebel and Oracle RAC Databases (11.2.0.4) from Exadata Cloud @ Customer (ExaCC – aka ExaCM) Data Pump import of the database exports for Siebel Download, Install and Configure Siebel 8.1.1.11(Innovation Pack 2013) / Patch 17 Install Siebel Tools and Client and Loadrunner scripts on Windows VMs Install Load Balancer for the 8 Siebel Application Servers Perform functional unit tests (Phase I) Test and Validate Load Runner Scripts Execute Loadrunner tests Compare results with baseline from AA Stage environment

Success Criteria Description Success Criteria Assessment Parameter Application Performance in Oracle Cloud @ Customer – Current State All Siebel application Performance to be as good or better than current Super Cluster solution Peak load test results Application Performance in Oracle Cloud @ Customer – 100% growth Peak load test results under a given volume growth of 100%

Program Agenda 1 Siebel Systems & Current Infrastructure Cloud Objectives & Scope POC Environment, Setup and Steps POC Results & Findings Cloud Next Steps 2 3 4 5 Confidential – Oracle Internal/Restricted/Highly Restricted

Choice and Control of Deployment Deploy workloads where you need them SELLERS: When discussing C@C, it must be clear that the compatibility is with OCIC. Same PaaS and IaaS hardware and software as Oracle Cloud Infrastructure Classic (OCIC, not OCI) Managed by Oracle and delivered as a service behind your firewall Same cost-effective subscription pricing model as Oracle Cloud Helps conform to business and government security requirements CAUTION SELLERS: most Apps Unlimited apps can run on approx 60 cores and the current minimum for Oracle Cloud Machine is 204 cores (a smaller footprint is on the roadmap) – so take care while proposing this option and make sure the customer has additional workloads that will run on some percentage of the remaining cores. Cloud at Customer It is running the exact same cloud software, and exact same and PaaS and IaaS services and is delivered as a service to your data center. And customers consume this as a service just like Oracle Cloud. Customers are not purchasing the hardware, they are not purchasing the software. But rather they are subscribing to the hardware, the software and the cloud operations. And Oracle managed all the operations for you – whether it is on-premises or off-premises. Oracle provides the customer with installation, configuration, patching, lifecycle management, upgrading and monitoring. As a customer, you have the choice as to where you want to deploy your services – public cloud or private cloud. Customers get the same user experience, the same capabilities, the same architecture and the same open standards that are available in the public or private cloud. We provide a fully-managed service behind your firewall. IaaS services are included in the subscription cost PaaS services are optional and additional cost Traditional Infrastructure On-premise Cloud@Customer Public Cloud Cloud Machines IaaS, PaaS Customer Data Center Purchased Customer Managed Oracle Cloud Subscription Oracle Managed

POC High Level Set Up Oracle Solution Center Oracle Cloud @ Customer (aka OCM) Oracle Solution Center American Airlines User Exadata Cloud @ Customer (aka ExaCM)

AA Siebel Sales POC Environment: OCC/EXACC Oracle Cloud @ Customer – OCM Model 288 288 cores Only 88 cores/OCPUs used across 11 VMs for Siebel App and Web tiers 2 TB Memory 840 GB used across 14 VMs 160 TB Storage Exadata Cloud @ Customer – ExaCM 1/4th Rack 2 Nodes – 22 OCPUs per node 44 Cores/OCPUs total 720 GB RAM per node 84 TB Usable Storage 10Gb Note: The Siebel on OCM/EXACM POC was conducted at the Oracle Solution Center in Santa Clara, CA

Compute nodes side by side comparison SPARC Super Clusters OCC/ExaCC (aka OCM/ExaCM) Application Domains 2 x 16 = 32 core SPARC T5 Processors (256 VCPU) 1 TB RAM 80 TB ZFS Storage Appliance Data Base Domains 2 Nodes - 16 OCPU per node SPARC T5 3.6 GH Processors (256 VCPU) 4 Exadata Storage Servers (144TB HC or 28.8TB HP) 12.8 TB Flash 3 x InfiniBand 36 port switches Oracle Cloud @ Customer - Model 288 88 OCPU (176 VCPU) 840 GB Memory used 160 TB Storage Exadata Cloud @ Customer - 1/4th Rack 2 Nodes – 22 OCPUs per node 44 Cores total used (88 VCPU) 720 GB RAM per node 84 TB Usable Storage 19.2 TB Flash

10GbE Network OCC (OCM) 10GbE Network ExaCC (ExaCM) External Non-OCC Windows Servers for Load Generator and Siebel Tools Simulator VM 1 to 4 10GbE Network Oracle Cloud @ Customer SOA Connection to On-Premise Oracle Cloud @ Customer (OCC – aka OCM) : IaaS Local Queues Local Queues Node 1 (Siebel Enterprise) Node 1 Siebel 8 OCPUs Siebel Enterprise Node 2(Siebel Enterprise) Node 2 Siebel Enterprise VM#9 Siebel 8 OCPUs VM#10 Siebel GW VM#11 Siebel 8 OCPUs Node 3 (Siebel Enterprise) Node 3 Siebel Enterprise VM#12 Siebel 8 OCPUs VM#13 Siebel GW VM#14 VM#7 VM#6 Web service Clients Siebel LB 8 OCPUs OCC (OCM) VM#5 VM#8 VM#15 10GbE Network Oracle Exadata Cloud @ Customer (ExaCC – aka ExaCM) : 2 Node RAC Node 1 Oracle DB EP RAC Node 1 Node 2 Oracle DB EP RAC Node 2 ExaCC (ExaCM)

November 10, 2018 American Airlines Siebel POC on OCC/EXACC Performance Testing Timeline and Flow POC Environment Provisioned Phase II Environment Ready for Testing Phase II Testing Complete DSC Load Runner Results DB Import / Creation Load Balancer Ready OCC/ExaCC (aka OCM/EXACM) Planning & Configuration Load Runner Validation Complete DSC Load Runner Results Application Setup Complete Load Test #1 Load Test #2 Load Test #3 1 Day POC Environment Creation 10 Days DB Import 1 Day Siebel SW Install & Config 11 Days Load Runner Validation 4 Days Phase II – Load Testing 2 Days POC Reports/Presentation 15 Days

Program Agenda 1 Siebel Systems & Current Infrastructure Cloud Objectives & Scope POC Environment, Setup and Steps POC Results & Findings Q & A 2 3 4 5 Confidential – Oracle Internal/Restricted/Highly Restricted

POC Performance Test Results Three (3) Iterations of Performance Load Tests were executed against Oracle Cloud@Customer environment. Comparison of Siebel Application performance response times were completed. Comparison was done between the Regression test execution and Production performance. Response times on the Oracle Cloud @ Customer for most applications show improvements. Few transactions show some degradation in the Oracle Cloud environment when compared with both Regression test baseline and Production performance. *** Regression test emulates Peak 100% volume, which represents the activity of regular daily Production volume. *** Note: After review, many of the delays in the Siebel transactions were attributed by: SOA connectivity back to customer’s external exposed stage URL site over the public network – NO MPLS connection was used as part of POC. The use of FastConnect/MPLS connection would resolve this delay in latency 5 of the 8 Siebel application servers were hovering around 97% memory utilization, which caused some excessive paging at peak times. (Apps Servers were slightly undersized with 30GB RAM using OC2M Shape.)

POC Load Runner Test Results 17.36% of transactions shows improved performance of 0.5 seconds or more 75.55% of transactions shows a change of +/- 0.5 sec ~7% of transactions shows reduced performance of 0.5 sec or more

Load Runner Test Results App # 1 Tests App # 2 Tests App # 3 Tests App # 4 Tests App # 5 Tests Member, Email, FNLN, Response Agency profile Improved Impact < 0.5 sec Degraded

Top Database SQL Comparison Examples AWR Reports - Elapsed Time (secs) * Note: The graph represents the same SQL statements found on the AWR reports generated from both current and SuperCluster DB environments

POC Database Monitoring – Enterprise Manager

Oracle Cloud @ Customer – CPU (per SAR reports) * CPU Usage never reached 33% for any of the Application Servers (8 OCPUs/each) Load Balancer / 2 Gateway Servers 8 Application Servers

Oracle Cloud @ Customer – Memory (per SAR reports) * Memory free on Apps Servers 2-7 may have been sized too low. (~97% utilization of 30GB) Load Balancer / 2 Gateway Servers 8 Application Servers

Oracle OCC/ExaCC (aka OCM/ExaCM) POC Summary / Results Successes: Better Performance Results OCC/ExaCC (aka OCM/ExaCM) performed very well overall. There were some transactions which showed degradation in performance, but most of them can be attributed to SOA connectivity back to American Airlines Data Center over public network and multiple URL redirects within the application to external URLs.

Oracle OCC/ExaCC (aka OCM/ExaCM) POC Summary / Results Successes: Ease and Speed of Provisioning Within a couple of days, the entire Siebel environment was recreated. The actual provisioning of the Siebel and database took few hours using the OCC/ExaCC (aka OCM/ExaCM) orchestrations.

Oracle OCC/ExaCC (aka OCM/ExaCM) POC Summary / Results Challenges: Since the POC was executed within Oracle Solution Data Center hence there was no MPLS (FastConnect) connections established between OSC and American Airlines Data Center to avoid the SOA connectivity related latency over the public network.

Overall POC Summary / Results

Program Agenda 1 Siebel Systems & Current Infrastructure Cloud Objectives & Scope POC Environment, Setup and Steps POC Results & Findings Q & A 2 3 4 5 Confidential – Oracle Internal/Restricted/Highly Restricted

Session : CAS1151 Next Generation Oracle platform for American Airlines’ Siebel Systems Session : CAS1151 Reza.Abouk@aa.com Mohammad.Khan@oracle.com