Abstract/Session Description: Session Title: Machine Learning Turbo-Charges the Ops Portion of DevOps Abstract/Session Description: Moving toward continuous (or short-cycle) delivery? Constantly rewiring your apps with microservice and similar architectures? How are you planning to maintain visibility and maximize service levels once this stuff gets into production? Coding instrumentation into your apps is time-consuming and error prone. Instead, let machine learning do the work of adapting your monitoring to your fast-moving application environments. In this session, we’ll discuss different types of machine learning that are optimized for operational data, and how they are leveraged to ensure your Ops moves as fast as the rest of your DevOps pipeline.

Machine Learning Turbo-Charges the Ops Portion of DevOps Oracle Code This is a Title Slide with Picture slide ideal for including a picture with a brief title, subtitle and presenter information. To customize this slide with your own picture: Right-click the slide area and choose Format Background from the pop-up menu. From the Fill menu, click Picture and texture fill. Under Insert from: click File. Locate your new picture and click Insert. To copy the Customized Background from Another Presentation on PC Click New Slide from the Home tab's Slides group and select Reuse Slides. Click Browse in the Reuse Slides panel and select Browse Files. Double-click the PowerPoint presentation that contains the background you wish to copy. Check Keep Source Formatting and click the slide that contains the background you want. Click the left-hand slide preview to which you wish to apply the new master layout. Apply New Layout (Important): Right-click any selected slide, point to Layout, and click the slide containing the desired layout from the layout gallery. Delete any unwanted slides or duplicates. To copy the Customized Background from Another Presentation on Mac Click New Slide from the Home tab's Slides group and select Insert Slides from Other Presentation… Navigate to the PowerPoint presentation file that contains the background you wish to copy. Double-click or press Insert. This prompts the Slide Finder dialogue box. Make sure Keep design of original slides is unchecked and click the slide(s) that contains the background you want. Hold Shift key to select multiple slides. Apply New Layout (Important): Click Layout from the Home tab's Slides group, and click the slide containing the desired layout from the layout gallery. Dan Koloski Senior Director Oracle Management Cloud March, 2017

This is a Safe Harbor Front slide, one of two Safe Harbor Statement slides included in this template. One of the Safe Harbor slides must be used if your presentation covers material affected by Oracle’s Revenue Recognition Policy To learn more about this policy, e-mail: Revrec-americasiebc_us@oracle.com For internal communication, Safe Harbor Statements are not required. However, there is an applicable disclaimer (Exhibit E) that should be used, found in the Oracle Revenue Recognition Policy for Future Product Communications. Copy and paste this link into a web browser, to find out more information.   http://my.oracle.com/site/fin/gfo/GlobalProcesses/cnt452504.pdf For all external communications such as press release, roadmaps, PowerPoint presentations, Safe Harbor Statements are required. You can refer to the link mentioned above to find out additional information/disclaimers required depending on your audience.

Program Agenda Defining terms 1 Defining terms Why (Dev)Ops is Perfect for Machine Learning Making Machine Learning Smarter Q&A 2 3 4

Program Agenda Defining terms 1 Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smarter Q&A 2 3 4

Defining Terms (source: wikipedia.com) Machine Learning Machine learning is the subfield of computer science that gives computers the ability to learn without being explicitly programmed. Evolved from the study of pattern recognition and computational learning theory in artificial intelligence, machine learning explores the study and construction of algorithms that can learn from and make predictions on data. DevOps DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. Systems Management or IT Operations Management IT Operations is responsible for the smooth functioning of the infrastructure and operational environments that support application deployment to internal and external customers, including the network infrastructure; server and device management; computer operations; IT infrastructure library (ITIL) management; and help desk services for an organization.

Program Agenda Defining terms 1 Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smarter Q&A 2 3 4

We have a problem: Dev has outpaced Ops Development is creating faster… Low-code Agile Microservices CI (Dev)Ops is promoting faster… Containers IaaS & PaaS CD Packages (the rest of)Ops is not moving any faster… #(*^(#^#)&^$(@^@($^ $(@)%&^$^**&^)!!!! …

One of Two Likely Outcomes, Both Bad OPTION 1: Your Changes Don’t Hit Production Until Ops is Ready Option 2: You Promote Unmanaged Code Anyway Confidential – Oracle Internal/Restricted/Highly Restricted

The Reason: Ops Depends on Human Effort Part of our problem is that the industry state of the art for many years has relied too heavily on human effort. We deploy 10s or 100s of monitoring silos and then expect humans to not only find the data, but understand it and know what it means, in time-compressed scenarios. The net result is that we are always behind, and “running to catch up.” Where’s the data? What does the data mean?

We Can Help! Ops Data is Perfect for Machine Learning Massive volume Highly patterned Predictable format Silos can be unified Seasonal trends Known sources Structured, Time-Series User Performance Metrics Server-side Performance Metrics (App & Infrastructure) Configurations Events/Alerts Transaction Payloads Unstructured Text Log Records It’s easy to see why IT organizations can’t keep up. They are drowning in monitoring tools and data, but have no insight. Previous generations of management tools expect that human intelligence will draw conclusions out of the data, but human operators are already overwhelmed with the velocity and volume of alerts and data coming their way, and so they get tired and miss things. Other tools have attempted to apply more advanced analytical techniques, but have only done so to subsets of the data because it’s too compute-intensive to do it across the board, so human operators are still required to stitch together information out the data silos. It’s time for a new generation of systems management.

Algorithmic Approaches to IT Ops Data Structured, Time-Series User Performance Metrics Server-side Performance Metrics (App & Infrastructure) Configurations Events/Alerts Transaction Payloads Unstructured Text Log Records Anomaly detection clustering It’s easy to see why IT organizations can’t keep up. They are drowning in monitoring tools and data, but have no insight. Previous generations of management tools expect that human intelligence will draw conclusions out of the data, but human operators are already overwhelmed with the velocity and volume of alerts and data coming their way, and so they get tired and miss things. Other tools have attempted to apply more advanced analytical techniques, but have only done so to subsets of the data because it’s too compute-intensive to do it across the board, so human operators are still required to stitch together information out the data silos. It’s time for a new generation of systems management. correlation prediction 13

Program Agenda Defining terms 1 Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smart for IT Ops Q&A 2 3 4

Maturing Machine Learning: A Three-Step Approach ML is not smart out of the box for every question To make ML smarter, know the questions you want to ask, then… Enhance Algorithms Increase Breadth Increase Depth

We Know The Questions We Want To Ask of IT Ops Data What caused the problem? How do I prevent the problem in the future? Is what I’m seeing normal or abnormal? What areas can I improve, and how? Most importantly, we know what questions we want to ask, so we can pre-tune the machine learning to find the answers. This is a very purpose-built set of machine learning, not just general data science. How is this application actually built/architected? What do I need to pay attention to right now? How should I rebalance workloads? WHAT WILL HAPPEN TOMORROW?

Working Example 1: Enhancing Anomaly Detection Begin with the Basics Distribution Based Unseasonal Model Daily + Weekly Additive Holt- Winter Modeling Automatic Season Detection Tune Based on Validation Robust to Sparse Pattern Variability Robust to Small Anomalies Graceful Transition from Daily-to-Weekly Evaluation Model Segmentation Daily seasonality detected. Base lines are wide because metric has a weekly pattern. Weekly seasonality detected and base lines much tighter around the observed values. Anomalies b/c observations higher than expected. Anomalies b/c observations lower than expected. No seasonality detected.

9x False Positive Reduction With Seasonality Enhancements Before: Weekdays and weekends are allowed to be imbalanced. Before: Flagged as an anomaly due to load/measurement variability. BEFORE: Anomalies are out-of-band samples. After: Select days to keep weekday-weekend balance. Graceful Day-to-Week Transition Sparse Pattern Variability After: Computing baselines at higher scale (hourly, configurable) solves this problem. Small Anomalies AFTER: Anomalies are statistically significant out-of-band samples.

5x Performance Boost with Data Pipeline Segmentation Unseasonal, Daily, Weekly Models for Metric #1 Unseasonal, Daily, Weekly Models for Metric #2 Data Pipeline Base Line Model Cache

Working Example 2: Enhancing Prediction/Early Warning Begin with the Basics Robust Linear Regression for Unseasonal Automatic Season Detection Tolerance Intervals Tune Based on Validation Season Specific Trending- Uncertainty Regime Change Detection Seasonal Pattern Trending Temporal Weighting Traditional Linear Forecast Enhanced (Tuned) Forecast

2x Improved Forecast Accuracy with Enhanced Trending Common corner case of inferred negative trends of sparse high seasons with stable patterns. Incorrect trend b/c small set, sparse high early in week, & most near low. Better treatment is to trend the “pattern” not “samples”. Unsupervised seasonality sometimes selects subtle highs. BEFORE AFTER Oracle Confidential – Highly Restricted

Additional Accuracy Using Temporal Weighting Example: a common based approach to robust linear regression is Thiel-Sen Estimator Thiel-Sen can be enhanced with specific tuning that accounts for expected seasonality in the underlying (IT Ops) data set, for example: Business day vs after-hours Same business day, different weeks Frequency/periodicity of samples Un-Weighted Temporally Weighted Temporal Weighting

DEMO: Matured Machine Learning in Action

Increase Breadth With Data Unification & Normalization Application Performance Monitoring Security Monitoring & Analytics Infrastructure Monitoring Log Analytics Orchestration Compliance Oracle Management Cloud Data Store Norm is repo by repo projects: slow and incremental. By centralizing data, we are able to deliver ML driven features more quickly. Convert to Time Series (Clustering & Rollup) Base Lining & Anomaly Detection IT Analytics

Increase Depth With Context, Topology, & Domain Expertise Topology: Tells us where to look. Context: Forecasted SLA violation & observe divergent correlation. Domain Expertise: Allows to identify root cause.

DEMO: Enhanced Breadth with Unified Data, Enhanced Depth with Context

Key Takeaways DevOps depends on “Ops” speed matching “Dev” speed The DevOps problem is well-suited to machine learning BUT… Machine Learning must be matured Unified data and context increases the effectiveness of ML and analysis Confidential – Oracle Internal/Restricted/Highly Restricted

Program Agenda 1 Defining terms Why (Dev)Ops is perfect for machine learning Making Machine Learning Smarter Q&A 2 3 4

Cloud.oracle.com/management For More Information For more information agter OpenWorld, visit cloud.oracle.com/management and follow us on social media. Cloud.oracle.com/management #MgmtCloud @OracleMgmtCloud community.oracle.com/mgmtcloud