1. Best Practices at AT&T
Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite
Bryon Rickey
Ravi Kayarthodi
September 20th, 2010

2. Agenda Welcome and Introductions
Supply Chain Management (SCM) environment
Overview/highlights
Architecture
Partnership with Oracle to transform our Business
Key project highlights
Future initiatives
Solution Support Center (SSC) and on-site support
Q&A
Ravi Kayarthodi:
My name is Ravi Kayarthodi. I am a senior architect at AT&T. I am responsible for our Corporate Supply chain platform architecture; platform includes Oracle E-Business Suite on Oracle Real Applications Clusters, several other real applications clusters databases and middleware and monitoring tools. I am an active member of Oracle RAC standards board at AT&T
5 years ago, we migrated our ERP System database (formerly Cingular Wireless’ ERP systems) to RAC to meet the increased availability, fault tolerance and scale-up requirements that arose from acquisition of former AT&T wireless and hence applications system consolidation and optimization initiative we started. After migrating to 3-node RAC, we made this platform as a central nervous system for our Finance and Supply Chain operation, managing all of inventory and assets. Currently it is our platform where all of OneATT’s orders flows though as we show in the next couple of slides.
Personally, I have worked on Oracle Products for over 16 years. I worked for Oracle Corporation before I joined AT&T team. I am an engineer by education and I also have a graduate degree in management.

3. About AT&T
AT&T is one of the world’s largest Telecommunications companies
AT&T Global Headquarters is based in Dallas, TX
AT&T has approximately 290,000 employees
AT&T has more than 90 million wireless customers
AT&T has more than 16 million high speed internet subscribers
AT&T has more than 2.5 million uverse subscribers
Exclusive U.S. provider of iPhone 3G / 3GS / 4

4. AT&T SCM: Architecture highlights
Oracle E-Business Suite Centric
Scale-up with cluster database on very large infrastructure
High availability with Oracle Data Guard, Oracle GoldenGate, and site level redundancy
Central inventory/financial management; utilizes product hub, planning/forecast with ASCP/Demantra
Order orchestration and brokering with in-house developed Oracle based application system
Internet deployment for interaction with external suppliers
Management reporting/profitability analysis; procurement and spend analytics
Industry leading warehouse/distribution management systems

5. AT&T SCM Platform: Financial Integration
Both Network and Retail SCM - financial counterparts are integrated in the same platform: Shared AP, AR, PA, FA, and GL
Supply Chain transactions leverage same referential data and accounting rules
Financial integration of supply chain transactions are seamless and are recorded real-time in the general ledger
No need for reconciliation between sub ledgers
Ability to forecast financial implications based on SCM forecasts
Online financial budgeting validation on purchasing and requisitions
Analytical GL Cube and Inventory Cube

6. Results
AT&T has saved more than $500 million and has improved its purchasing power by combining accounts with its vendors.
Estimated to save $1B in inventory costs, and $300M per year in reduced operating costs.
Warehouses have been able to keep inventories low without missing SLA, resulting reduced overall inventory costs.
Ability to identify sold merchandise, resulting in fewer fraudulent returns

7. Industry Recognition 2010 IDG’s CIO Award
Recognizes organizations around the world that exemplify the highest level of operational and strategic excellence in information technology

8. Systems Architecture

9. AT&T SCM: E-Business Suite Centric
This looks very Complex, right? No, it is not so much complex but it does support complex business functions.
As you see in this diagram, all our business functions interact with this central system powered by Oracle E-Business Suite.
This central system is our system of record
It is our operational data store too
All the supply chain, asset management and financial function from the order capture to fulfillment, from procure to pay, and from acquire to retire are performed in this system
It provides integration with billing applications
It provides market making capability utilizing efficient supply chain configuration

10. AT&T SCM: Key Systems Architecture
EDI
EDI GL
ODP
eAuction PO AP OM
ASCP
RNF PA FA
Inventory Planning
Purchasing
Financials
Oracle E-Business Suite 11i
Receipts
Receipts
Audits
As we see in this slide, our application system provides links to customers & suppliers
Our system supports and integrates business processes
It links to supplier on the top and customers on the bottom via distribution center
These systems also provide inventory tracking and asset tracking ability
As we see here, demand planning for our assets and also for inventory, recording all the financial data in the same system without loosing sight into receipts and shipments.
Shipments
Moves
Repairs
Warehouse Mgmt
Inventory Tracking
Distribution Center
Corporate Asset Tracking

11. Oracle 11i E-Business Environment
Forward Orders
Store
Replenishment
Dealers/
iStore
National
Retailers
DSL and
uVerse HSIA
Direct Fulfilment
Upfront
EPL
Planning
Purchasing
Order
Management
Inventory
Receivables
Financials
Oracle E-Business Suite 11i Apps
Store
XBM/Insurance
Dealer/NR
Service Depots
RU30
Opus
Sales
Returns
Company Owned Retail
Reverse
In the order management – Forward orders from Store Replenishment, dealers, technician orders and forecasted demand gets into SCM platform for processing.
On the reverse order end we have Corporate Stores, returns, Service depots and so on
In summary, we have a database cluster having 416 CPU cores and 1.7TB memory in the DB cluster alone
On a typical day, this DB cluster processes 25 million DB transactions, over 200,000 concurrent jobs and over 2B SQL executions
416 CPU Cores & 1.7TB of Memory in DB Cluster
14TB of storage for transaction DB
3TB of storage for Archived DB

12. AT&T SCM Platform: Oracle10g RAC Infrastructure: VCS+ CRS
Highlights of this architecture – showing redundancies at network, storage and other infrastructure components level.
416 CPU Cores (104 per node from Sun/Oracle Enterprise 25K frame), and 1.7TB allocated memory
98 CPU cores in 6 Application tier servers
In the cluster, we have 8 counts of 1 Gigabit Ethernet for private interconnects per node 4 for UDP/cache fusion traffic while other 4 for shared storage (SFRAC) cluster interconnect
16 storage ports per server
Dedicated backup network
We have Oracle Cluster ware/ Cluster ready service and also utilize vendor provided cluster – VCS for shared storage and file system
We monitor interconnect traffic periodically
We have tweaked LMS processes to provide us optimal performance, LMON runs in real time priority

13. AT&T SCM Platform: Transaction Volume
Approximately 3.35 million logons to the production areas per day
2.06 billion SQL executions per day
Approximately Billion logical reads and 41TB of physical reads performed per day
Approximately 7 TB of writes performed per day
Approximately 1.05 billion user calls and 52 million transactions completed per day
200,000 concurrent request per day

14. AT&T SCM Platform: Operational architecture
Decoupled month-end book closing preparation
Statistics gathered on a clone
Separate reporting environment – refreshed nightly; Inventory and GL Cubes for reporting
Revenue share and allocation on a separate system
Direct supplier interaction with Internet-deployed application modules
Utilizes market making mechanisms (sourcing)
Preparing out system for month-end-closing
We gather statistics on the cloned environment and then import into production environment.
We do not allow scheduled reporting on this cluster database; instead we process of clone reporting from operational database nightly
Revenue share/allocation on separate system
Very same production environment had sourcing application for reverse auction that is time critical

15. Oracle 11i ERP/SCM– Reporting environment Refresh Summary
Nightly reporting database refresh
Nightly Asset Synch and Audit summarization from PA, FA, AP,PO,CATS
Network SCM reporting data to external System
251 Stored Discoverer reports with over 21,000 folders
Business Objects reports (Ad-hoc and out-of-box)
Reporting environment is available for summary and detail reporting. Complete set of data is available as of previous day 11:00pm
Asset
Tracking
Asset Synch
ERP PROD
Corporate
Warehouse
Data Guard
Data Guard
L-TRA
Shadow Image
RPT
External
System
Archived
Transaction
History
Slide 15:
Nightly reporting environment build goes through the shadow image synchronization and slit process.
First, we have 2 tiers of recovery environments for business continuity; LTRA (providing database level redundancy) and RTRA (providing site level redundancy) in our deployed architecture
Reporting environment build uses LTRA and Hitachi shadow image tool
Every night our reporting build job stops this reporting database and starts synchronizing storage blocks with LTRA storage without stopping LTRA; once block level synchronization is completed, it breaks that mirror, opens the database and completes summarization.
We utilize transportable tablespace to keep historical summary allowing summarization process only for new records.
We use reporting database use for detailed reporting, and we use discoverer, business objects and other ad-hoc tools
Reporting environment available for users 2 hours
BI Discoverer
BOBJ

16. AT&T SCM: Oracle 11i Dev/Test Configurations
Bothell, WA
HDS – USP-V
HDS – USP-V
Alpharetta, GA
NetApp
GOLD
R-TRA
L-TRA
PROD
Data Guard/
Custom Shipping
Data
Guard
Data Guard
Data Guard/
Custom Shipping
Shadow Image
MET2
NetApp
GOLD
Shadow Image
NetApp
Master
SnapMirror
RPT
FTP
NetApp
Master
Flex Clone
Flex Clone
Shadow Image QA
DEV
DEV QA
DEV
Flex Clone
With several simultaneous development projects running in parallel, we have several copies of production environment to support these projects. We apply strategy to supports these projects.
We use NetApp Flex-Clone/Mirror tools to clone Development/test environments quickly. We are able to complete environment clones in 2 hours.
We use traditional cloning methods to clone Performance test environments that need to be in SAN/fiber attached storage.
MET1
DEV
DEV QA
Flex Clone
FTP
DEV
DEV
DEV QA
Storage Efficiency with NetApp de-duplication Tools
Performance test environment on tier-1 storage

17. AT&T SCM: Oracle Automatic Storage Management
Local and Remote DR databases and also Reporting databases have been migrated to ASM storage to meet storage and capacity needs
Avoided SFRAC costs
Performance
SCMP DR/BC database converted to RAC database
Saved SFRAC license cost on E25K frames
Comply with AT&T standard for cluster database (ASM is required)
Expected to save cost in the future when we upgrade other Corporate Systems’ databases
Since we started with Oracle9i RAC, having stabilized and mature environment performing optimally, we continue to use SFRAC in our production database. However, our standard is to use ASM with RAC. We have migrated business continuity environments on RAC and have avoided costs. With this we are ready for ASM

18. AT&T SCM: Archive and Purge
Archive Stats
16 custom and core modules including (GL, INV, OM, AR, WSH, PO, FA)
Executed weekly, monthly, quarterly, and yearly
History instance
Informatica Data Archive application used
Schedule is monitored real-time to ensure adherence
Purge Stats
70 jobs via Oracle concurrent programs
Executed daily and weekly
To keep our production operation database in trimmed to achieve better performance, we archive older transactions. We apply aggressive archive schedule yearly, quarterly, monthly and weekly archive strategy and store that in the database that can be accessed for reference both from production and reporting environments. We also use enterprise archive with compression achiving storage efficiency.

19. AT&T Supply Chain Geographic Distribution – DB Flow to Test and Dev
Primary Site XX XX XX XX
Test Site 1 XX
Development
Site 2 XX
Test Site 2 XX XX XX XX
As we see in this picture, our application environments spread into six data centers, located various part of the country. We have distributed considering both disaster recovery and proximity to user group groups
DR Site
Development
Site 1 XX XX XX XX
Prod XX XX XX XX XX XX XX
Dev/Test XX XX XX XX XX XX

20. AT&T SCM: Spend Analytics
Oracle Business Intelligence Enterprise Edition (Oracle OBIEE) Powered
Sourcing from Oracle E-Business Suite and Legacy Applications
Top 5 Categories
Spend
Top 5 Suppliers
We utilze OBIEE for our Supply chain procurement and spend analytics, extracting data from operation database using data guard. This fulfills our business users to analyze spend metrics and contract leakages and so on.
Line Of Business
Contract Leakage
Off Contract Spend
Payables Leakage

21. AT&T SCM: Database Maintenance/Performance Management
Monthly month-end close of books stats validation – database statistics preparation on production clone
Table Re-organization – routine and ad-hoc
Periodic index rebuild
Daily performance call
Daily/weekly/monthly/yearly purge/archive schedules – vigilance in adhering to them
Extensive performance monitoring on all levels of the system
In summary, this architecture provides operational performance we need with our performance management tasks like
Decoupled DB statistics management for month book close jobs.
Periodic and ad-hoc table reorganization and index rebuilds
We have daily performance calls to analyze and discuss performance issues
Extensive performance monitoring.

22. Partnering with Oracle Advanced Customer Service
Oracle Solution Service Center (SSC)

23. SSC: Project Spotlight: Patch Recommendations
Project Activities
Description
Identification of available latest required patches for AT&T environment
An initiative to incorporate E-Business Suite recommended patches, such as, Rollup (RUP) in AT&T’s release schedules for 2010
Participation in meetings to review recommendations
Value Addition
SSC and AT&T worked together to create a new process to efficiently identify required patches with the goal to minimize the impact of rolling them out in our production environment and to current/future projects:
Quarterly document for AT&T environments with the latest recommended patches.
Participation on AT&T IT meetings to clarify recommendations and questions
Provide detailed patch analysis on a ad-hoc basis
Successful roll out of AP.O patchset along with other Financials patches in February 2010: stabilized production environment and brought environment to the latest code base.
Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application Technology Group (ATG) RUP-7 for the April 2010 production rollout. 23

24. SSC: Project Spotlight: Performance Improvement
Project Activities
Description
Understand AT&T batch window requirements and transaction volumes
An initiative to address various performance issues related to Month-End-Close processes and other batch processes after Oracle 10g database upgrade.
Identification and root cause analysis of problem areas
Implementation of resolutions
Value Addition
Active participation on Conference bridge calls and OWC sessions.
Engage Oracle development groups (Financials and Apps Performance) to provide SQL tuning improvements.
Improve month end close activities like AR close, depreciation and asset retirement, optimizer, Invoice Validation processes.
Maximize performance during month end, preventing delay; AT&T was able to successfully meet the SLAs for the Month End Close Processes month after month.
Provide recommendations for major upgrades (DB, Applications and Tools)

25. SSC: Critical SR Spotlight: Other Key areas of focus
Project Activities
Description
Workflow background process
WF_DEFERRED queue growing exponentially; AT&T engages SSC to evaluate and address the issue.
upgrade.
RAC – CRS Configuration
Collect diagnostic and configuration data
Troubleshoot frequent RAC node reboot/performance issue post upgrade
Discover unforeseen problems
Validate AT&T configuration with RAC Best Practices
Device Life Cycle Database Upgrade
Provided upgrade expertise and support (pre and post deployment for 11g R2 upgrade.
Value Addition
Active participation on conference bridge calls and OWC sessions
Review current setups and schedules of workflow Background Process
Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg. skip script)
Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED queue
Monitor the implementation of recommended suggestions
Engage Oracle Development and RAC COE team
Engage on Architectural design discussions
Stand-by support for DB/RAC upgrade deployment activities
Ensure successful implementation of RAC upgrade Rollout

26. SSC: AT&T Targeted Services: Operations
The Services Yield to the Customer targeted for the next 6-month planning period is comprised of key Support activities, aligned with key customer objectives/activities towards achievement of AT&T’s Account Goals
Supply Chain Transformation (Streamline Operations for newly deployed modules)
Work with ACS sales to determine appropriate support for upcoming SC rollout. Determine/secure funding for new modules (Demantra and Transportation Management Suite)
Develop, Enhance and Secure development sponsors
Participate in scheduled project “Deep Dive” calls with PMO and project team
Site visit to Memphis to understand warehouse and logistic operations – understand business functions.
Supply Chain Business – develop key relationships with Retail and Network business users
11i Financials and Supply Chain Production Supportability (System)
Next generation iPhone launch and 2010 Holiday Retail Season readiness
“Top Ten” Prioritization
Future – implement Tactical vs. Strategic

27. SSC: AT&T Targeted Services: Transportation Manager
OneATT- New Horizon to transform Network and Asset Management: Logistics TMS
Key Secondary Activities
Participate in scheduled project “Deep Dive” calls with PMO and project team
Work with PMO/SSC to determine appropriate support model for New Horizon’s
Work closely with OCS /PMO on project milestones – develop relationships
Continue to maintain Dev OTM Network
Supply Chain Business – develop key relationships
Site visit to Warehouses
SSC walk-thru’s
Key Deliverables
Enhance Supply Chain SSC presence and understand usage
and architecture for OTM
Site visit to understand implementation, document in
configuration guide.
Performance testing as requested.
Amend SSC contract to include OTM products
Future – implement Tactical vs. Strategic
OneATT initiative to consolidate Supply chain system - New Horizon to transform Network and Asset Management:
Project - Logistics TMS

28. SSC: Oracle E-Business Suite Release 12/Database 11g R2 Upgrade
Upgrade existing Oracle 11i.10 SCM and Planning environments to E-Business Suite R12. This will enhances supportability for our complex environment and provide additional needed functionalities
Three stage upgrade (2 – 3 year journey)
Supply Chain planning to release 12
RDBMS for 10g to 11G
Supply chain platform to release 12
Objective:
System Availability, Supportability, and Software Currency
Key Secondary Activities
SSC “Top Ten” Prioritization: Service Request review meetings with Supply Chain, Financials, and DBA support teams
Patching/Release Management strategy for Release 12 interoperability;
Leverage Tools and Templates
Daily SSC Performance Calls
Key Deliverables
Proactive Diagnostics (Configuration/Performance)
Contracted deliverables
Semi-Annual Account Review
Future – implement Tactical vs. Strategic

29. Solution Support Center
Service Highlights
Focused Engineering Team of Oracle Subject Matter Experts
Fast track solutions based on intimate Knowledge of customer Environment and business needs
Proactive Software Advice and Knowledge Transfers
Performance and Patch Assessments
Single point of contact for all technical issues - Technical Lead
Single point of contact for all escalations – SDM
24/7 coverage
Value
Mitigate Upgrade Risks substantially
All project milestones are met
Ensured optimal usage of Oracle features and functionality
Reduced overall Maintenance/upgrade/ training costs

30. Solution Support Center
Key Delivered Features
Focused Engineering Team
Single Point Accountability – SDM
Exclusive number 24x7x365
Review of Critical Issues as Needed
Patch Analysis
Stand-by Support for Critical Go-lives
Proactive Alerts / Patch Notifications
Project Planning / Migration Assistance
Periodic Review Sessions
Applied Research

31. Questions

32. Ravi Kayarthodi (RK8409@att.com).
For More Information
On this presentation, Bryon Rickey or
Ravi Kayarthodi
For general information on Advanced Customer Services, go to
oracle.com/acs or