Optimizing Oracle Exadata with Oracle Support Services [CON7054] A Client View from KPN Eric Zonneveld Architect Engineered Systems Oracle Advanced Customer Support Jan van Dijken Service Delivery Engineer October 02, 2014

Program Agenda 1 KPN Overview Platinum Services Advanced Monitoring and Resolution Advanced Customer Support Lessons Learned 2 3 4 5

“We have multiple suppliers who deliver support and updates, but that makes it difficult for us to meet our service levels. We believe Oracle Platinum Services can improve the way we support our environment so that we are able to meet our service levels and, at the same time, lower our costs.” Play video Michel Bosman Lead Architect, KPN

KPN Overview Infrastructure, project and organization

KPN Overview Project 2010-2011 2012-2013 2014 Project startup (Mar) Ordering (Nov) POC Systems delivered X2-2 quarter (Dec) Phase I: Initiation and testing (Jan) Phase II: POC and pilots (Feb) Phase III: EXA demo (Mar) Phase IV: DTAP systems delivered X2-2 full (Jun) First critical application chain live (Nov) 2012-2013 Platinum Services (Aug) DTA systems delivered X3-8 full (Jan) Oracle ACS and AM&R contract for service delivery (IM, PM, CM) (Jan) EXATEAM created (Jan) Migration DTA -> X3 (Jun) Separate production and non-production at rack level 2014 Delivery second production P2 based on 12c (Jan) Migrate workload P1 to P2 (Apr) Exalogic (May) Exalytic (Aug)

KPN Overview Infrastructure Haarlem Aalsmeer 40 km Dataguard

KPN Overview Total Hardware 24 compute nodes 62 storage nodes 560 cores 10,8 TB memory 55,8 TB flash 1,824 TB storage 16 IB switches 6 CISCO switches 4 KVM Per Site Storage bandwidth 50GB/s X2 93GB/s X3 per rack 1,5M IOPS (read) 1M IOPS (write) per full rack

KPN Overview Interesting landscape … Interesting objectives … Challenges Interesting landscape … Huge Oracle database landscape (>2,000), lots of Oracle middleware and applications All hardware and OS platforms in 27 datacenters All versions (e.g. Oracle database 7 and up) Interesting objectives … Data center migration and application / chain consolidation Company strategy is to have vendor / partner take more responsibility Decrease spend

KPN Overview Which leads to … While at the same time … Challenges Which leads to … Increasing workloads because of consolidation Increasing demands for performance, stability and availability While at the same time … Aggressive outsource strategy Continuous re-organization Challenging indeed

KPN Overview “Old fashioned” support did not work “New way” of working How to engage “Old fashioned” support did not work Tiered model Offshore “New way” of working DBA centric, end-2-end responsible 1 team onshore Strong combination between Oracle (run) and KPN (change)

KPN Overview Oracle Platinum Services Organization Oracle Platinum Services Change management (Patching) Oracle Advanced Support Cloud Services Incident management, problem management Oracle Onsite Support Innovation, life cycle management (infra upgrade) KPN ITS Platform Business Applications Change management, business consulting, backup & recovery

KPN Overview Scalability Performance Benefits of introducing Exadata Servicing 594 databases (incl. 132 standby databases) 220TB storage used (net) 58 applications Performance Performance gains ranging from 2x to 100x Special hardware and backup infra no longer needed Application scaling up 10x volume without a problem

KPN Overview Availability Faster life cycle management Benefits of introducing Exadata Availability Availability increase (no unplanned downtime) Real Application Clusters Active dataguard Maximum availability architecture Faster life cycle management Most up-to-date and secure platform Introduction of 12c at least a year before other platforms

KPN Overview But what about support? Benefits of introducing Exadata Incident Management SLA’s (notification, diagnostic, resolution) MTTR Service request handling Problem management Business KPI’s (anything)

Platinum Services Make the invisible visible

Platinum Services Part of hardware support “Free” but with rules What is it? Part of hardware support “Free” but with rules Limited scope for incident management Oracle Advanced Support Gateway server OEM12c, customized with metrics, thresholds and escalation rules Patching Change management (in ACS portal)

Platinum Services Visibility of incidents (not events!) What changed? Visibility of incidents (not events!) Visibility of problems Visibility of (wrong) configuration RDS in Oracle executable Visibility of Oracle best practices not being applied Huge backlog Moved from >120 open SR’s after engagement to average 15 SR’s currently

Oracle Monitoring Advanced Support Gateway Platinum aids company DBA’s with the incident management process

Oracle Advanced Support Cloud Services Advanced Monitoring and Resolution

Oracle Advanced Support Cloud Services What is it? Extends Platinum Services, adds problem management Broader scope More events Diagnostics RCA’s Resolution Integration with client IM/PM processes (ITIL-based) Service Now

Oracle Advanced Support Cloud Services What changed? More responsibility delegated to Oracle (IM and PM process) Shifts focus from KPN DBA’s from running the infrastructure (run) to aid applications and business processes, introduce new functionality (change) Advanced Support portal

Oracle Advanced Support Cloud Services What changed? Increased service level objectives, e.g. Resolution Severity 1: 80% < 1 day, MTTR average < 1 day Severity 2: 80% < 3 days, MTTR average < 3 days Availability reporting Of (by the client) nominated critical database services and related applications, >99.99% Incident summary reporting, quarterly Production / non-Production, by date and severity, by category (Exadata software, database, compute node, storage, switch, other)

Oracle Onsite Support Customer Integration

Oracle Onsite Support What is it? Acts as the “glue” between Oracle Platinum, AM&R and the client Life cycle management (upgrades) Innovation New functionality, new releases Best practices, Standardization Expert troubleshooting On site, close contact, customer intimacy

Lessons Learned 2 years of engagement, what worked and what didn’t?

Lessons Learned Oracle should be more visible in what they do Room for improvement Oracle should be more visible in what they do Frequent face-to-face reporting in a format that appeals to the customer Client does not want to know the differences within Oracle (Platinum, AM&R and ACS), should be one team and one voice Incident to problem promotion can be improved Portal could be improved Google search and user friendliness CI tagging

Lessons Learned Continuous learning within AM&R / Platinum Room for improvement Continuous learning within AM&R / Platinum Enhancement of KA’s (knowledge articles) must become second nature Change management Integration of Oracle changes into the CM system and processes of the client Work on “awareness” of customers of patch frequency, hardware maintenance etc. Communication between support engineers and customer Do not assume customers read all mail or plough through the portal every day

Lessons Learned What worked? Insight into Exadata significantly improved (exposure of problems) Co-operation between KPN and Oracle Service delivery management (TAM) Co-engineering between Oracle and KPN Local ACS resources Complex problem solving

Oracle Services Advanced Customer Support and Support + + =