Execution Monitoring Notification Aychin Gasimov DOAG 2016

Pros Cons Enterprise Manager Cloud Control Wide range built-in metrics Reach functionality Includes job system Incident management and reporting Not just monitoring but also management Pros Configuration Complexity BUGs and week support Cons

Database can send incidents and problems saved into ADR to Enterprise Manager Incident Rule Set Check Apply threshold Generate event New Event Incident occures Store in ADR Send to EM Diagnostic Incident

Define metrics Generate Event Process Event

Design flow Admin Groups Templates Event Rules Jobs Credentials

Define hierarchy and create Admin Groups Templates Event Rules Jobs Credentials Define hierarchy and create Admin Groups Groups targets by their type and properties Targets become the member automatically Automates target management on subgroups level Each entity is separate admin group ADMGRP Production Database Instance Cluster DB Host ASM Cluster ASM Mission Critical Development Staging Test Lifecycle Status Target type

Create Monitoring Templates and Template Collections Admin Groups Templates Event Rules Jobs Credentials Create Monitoring Templates and Template Collections One template per target type and lifecycle status Centralized metric management Starting from 12.1.0.4 define Adaptive Thresholds Group Monitoring Templates to Template Collections Associate Template Collections with Admin Groups Template Collection for Prod ADMGRP Production Database Instance Cluster DB Host ASM Cluster ASM Mission Critical Development Staging Test Lifecycle Status Target type

Create Event Management Rulesets Admin Groups Templates Event Rules Jobs Credentials Create Event Management Rulesets Associate with Admin Groups Define Target availability Metric alert Metric Evaluation Error Define rule for notifying about Problems (from ADR) SMS notification for Mission Critical and Production targets Create incidents for specific conditions System Response time Development Mission Critical Create Incident Send email Capacity Performance Send SMS and email New event Lifecycle Action

Create Jobs Use Jobs to execute OS scripts Create Jobs in Job Library Admin Groups Templates Event Rules Jobs Credentials Create Jobs Use Jobs to execute OS scripts Create Jobs in Job Library Define them as “OS Command” and “Command Type” as “Single Operation” Define E-mail notification In Job definition (notify only owner) In Incident Rules Submit from library to activate as required Associate with Admin Groups Associate with Admin Group

Central location for scripts and losg Admin Groups Templates Event Rules Jobs Credentials Create Job to sync central folder over enterprise Maintain one place to manage all scripts Synchronize central location to local hosts Benefits: Ease of maintenance High availability High redundancy Apply the Job to the highest level of the Admin Groups Execution logs can be synced back to central folder Management host /home/oracle/scripts /home/oracle/logs Management host /home/oracle/scripts Central location for scripts and losg /home/oracle/logs Management host rsync -a --delete user@host:SOURCE DEST /home/oracle/scripts /home/oracle/logs

Define credentials for the targets Admin Groups Templates Event Rules Jobs Credentials Define credentials for the targets Create Global Named credentials for MC and Production databases Use same global credentials for Primary and Standby databases as default ones Define “Database Host Credentials” for Database Targets, they used by Jobs If some target types mostly use same passwords, then define default credentials for them

Adding new target to EM Cloud Control Save target Set credentials Set Lifecycle Status Lifecycle Status = Production Contact = BOTH Database Instance Template Collection Cluster DB template Host template DI template Prod-Grp Jobs on Prod-Grp Incident Ruleset for Prod-DI-Grp

General recommendations Use corrective actions, specify them in templates To fix issues To gather additional information Wait some time after configuration to filter out useful metrics and adjust thresholds Check already generated incidents and add them to your rules if required Backup important components periodically and before major changes. Use emcli utility with next options: export_admin_group export_incident_rule_set export_jobs export_metric_extension export_template Backup OMS configuration (emctl exportconfig)

Availability problems Monitoring should be uninterruptable Scripts should be able to run without interruption EM 13c provides new Always On Monitoring (AOM) Configure Agent Out-of-Bound notification Job executions can be offloaded to local crontabs AOM Target Availability events AOM Scripts running over Job system Metric Alerts Custom Solution

Custom solution for Jobs availability Custom script resync_em_to_cron.sh Extracts data from EM repository Converts it to crontab format Appends to crontab Create EM Job which calls the above script with “add” option Create EM Job which calls the scripts with del option EM Jobs DEPLOY_JOBS_TO_CRONTAB UNDEPLOY_CRONTAB_ENTRIES EM Repository DB … add %TargetName% … del resync_em_to_cron.sh hosta resync_em_to_cron.sh hostc resync_em_to_cron.sh resync_em_to_cron.sh hostb hostd

https://aychin.wordpress.com/downloads Questions Questions https://aychin.wordpress.com/downloads