OMS & Repository Management EM-Season Ep.3

Who am I? (Deepak Sharma) ORACLE DBA for 5 years. Worked on technologies like Oracle 11g Grid Control, Oracle 12c Cloud Control, Oracle Configuration Manager PeopleSoft SQL Server Oracle Certified Professional. Active Speaker of AIOUG-NIC OMS & Repository Management

EM12c Series Topics… New Features in 12c cloud control EM12c Cloud Control Architecture Installation & Configuration OMS & Repository Management EM12c console, S/L, patching & provisioning Best Practices of Monitoring Plug-ins & Metrics Extensions Creating & Managing Jobs in EM12c Incident Management Backup, Recovery & High Availability.

Oracle Enterprise Manager Overview Centralized Monitoring, administration and Lifecycle Management Oracle and Non-Oracle Technologies IBM Websphere MS SQL Server MySQL Etc. Complete IT Infrastructure Oracle Enterprise Manager has been rebranded from Grid Control to Cloud Control OEM 12c Cloud Control New Features

Oracle Enterprise Manager Overview Components Oracle Management Agent Oracle Management Service Oracle Management Plug-ins Enterprise Manager Cloud Console Oracle Database for Management Repository OEM 12c Cloud Control New Features

OEM 12c Architecture OEM 12c Cloud Control New Features

Contents {Today’s Topic} Oracle Management Agent Oracle Management Service Oracle Management Repository Troubleshooting OMS & Repository Management

Oracle Management Agent Prior to 12c Oracle Agent had to be managed from Command Line. Either from EMCTL or EMCLI In 12c Oracle Agent can be managed easily from OEM12c Console. OMS & Repository Management

Agent Management from Console Navigation Setup -> Management Cloud Control -> Agent This pages displays status of all the agents configured Status will include “UP”, “DOWN”, “UNDER”, “BLACKOUT”, “UNREACHABLE”. Operations like “Start Up”, “Shut Down”, “Restart” etc can be easily perform by selecting a particular agent. OMS & Repository Management

Agent Management from Console OMS & Repository Management

Agent Home Page OMS & Repository Management

Agent Upgrade from Console Key Notes Upgrading OMS does not Upgrade Agents. Agents can be easily upgraded from the console. How? Setup -> Manage Cloud Control -> Upgrade Agent Click “Upgrade Agents” under “Agent Upgrade Tasks” Click Add to add the agents to be upgraded and submit the job. After Upgrade perform “Post Agent Upgrade Task” OMS & Repository Management

Agent Upgrade OMS & Repository Management

Agent Upgrade OMS & Repository Management

Managing Multiple Agent One of the major enhancements in EM12c. Multiple agents can be managed at one time. This feature saves a lot of time. Eg. We have to change “upload interval” for agent Select multiple Agents you want to modify Click “PROPERTIES” icon This will start “Agent Configuration Operation” Job Select multiple agents Click Parameters Tab. Change Property Value and click Submit. OMS & Repository Management

Example OMS & Repository Management

Example OMS & Repository Management

Example OMS & Repository Management

Using Agent’s Homepage HOW? – Click on the Agent Name. OMS makes connection to the agent and displays variety of information. Like:- Summary Region Monitoring Region Performance & Usage Region Resource Consumption Region Incident Region OMS & Repository Management

Example OMS & Repository Management

Agent Homepage Navigation to some interesting and useful pages: Agent Menu Monitoring Diagnostics {Support Workbench} Control {Start/Stop Blackout, etc..} OMS & Repository Management

Example OMS & Repository Management

EM Agent Browser This GUI which was developed several years ago. OMS was being built during that period. This feature was disabled in 10g and 11g. Re-introduced back in 12c. URL: https://<agent_hostname>:<agent_portnumber>/emd/browser/main OMS & Repository Management

Example OMS & Repository Management

EM Agent Browser Function Takes system state dump for the agent. Useful for debugging problems. How to use? Click on “System State Dumps”. Click “Perform System Dump”. XML file is generated. OMS & Repository Management

Example OMS & Repository Management

Agent Management with Command Line Command Line Interface types: EMCTL EMCLI EMCTL Utility: Is used to control different modules Can be used from Agent’s Homepage emctl getemhome OMS & Repository Management

Some Important Commands Emctl start agent Emctl stop agent Emctl status agent Emctl clearstate agent Emctl secure agent Emctl pingOMS Emctl getversion agent Emctl listplugins agent OMS & Repository Management

Create Blackout Example >emctl start blackout <blackout_name> [<target_name>[:<target_type>]] … [-d <duration>] (nioug) (/app/oracle/product/agent12c/agent_inst/bin) oracle@houtestdb1 $ ./emctl start blackout test houtestdb1.test.domain.com:host -d 09:00 Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Blackout test added successfully EMD reload completed successfully Emctl Logs Directory {emctl.log}: $agent_home/agent_inst/sysman/log OMS & Repository Management

Blackout Example OMS & Repository Management (nioug) (/app/oracle/product/agent12c/agent_inst/bin) oracle@houtestdb1 $ ./emctl status blackout houtestdb1.test.domain.com:host Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Blackoutname = test Targets = (houtestdb1.test.domain.com:host,) Time = ({2015-01-29|08:58:22|540 Min,|} ) Expired = False oracle@houtestdb1 $ ./emctl stop blackout test Blackout test stopped successfully EMD reload completed successfully oracle@houtestdb1 $ ./emctl status blackout houtestdb1.test.domain.com:test No Blackout registered. oracle@houtestdb1 $ OMS & Repository Management

EMCLI Enterprise Manager Command Line Utility. Used for automating commands, to be executed against large number of targets. Saves time, scripting is much faster than using EM Console for each target. OMS & Repository Management

EMCLI Example Update life cycle property “CONTACT” for 500 targets. Group Name: ProdDB >emcli set_target_property_value –property_records= “ProdDB:composite : Contact:dksharma”-propogate_to_members >emcli add_target_property –target_type=”oracle_database” –property=”Environment” This changes CONTACT detail for all targets under ProdDB group to “dksharma”. OMS & Repository Management

EMCLI Architecture EMCLI Client OMS Extension {installed within OMS} Client ------(request)-------- > OMS Extension OMS Extension -------(passes)----------- > OMS Can be installed on any system. Setup -> Command Line Interface OMS & Repository Management

Agent Log & Trace files AGENT_BASE_DIR :- prompted while installation. Agent Inst Dir Mgmt Agent Home Home (Installation Files) Agent Related Configuration Files OMS & Repository Management

Log & Trace files location Agent Heartbeat History <agent_inst>/sysman/log/gcagent.log Agent & Watch Dog Startup <agent_inst>/sysman/log/emagent.nohup Trace Files <agent_inst>/sysman/log/*.trc Emctl command output <agent_inst>/sysman/log/emctl.log Error while securing Agent <agent_inst>/sysman/log/secure.log OMS & Repository Management

Oracle Management Service Web-based Application Communicate with Agent and OMR to collect & store information about all the targets Can be controlled via both EM12c Console & command line OMS & Repository Management

OMS Management from Console Navigation Setup -> Manage Cloud Control -> Management Services OMS & Repository Management

OMS Management from Console Run time behavior of the OMS can be alter. How? – Setting Properties as needed Click on “Configuration Properties” By default all the “Non-Default” properties are listed. OMS & Repository Management

Example OMS & Repository Management

Management Services Homepage Click name of OMS to see more details. OMS & Repository Management

OMS Menu OMS & Repository Management

Management Services Homepage Must be wondering ??? Start, Stop is not listed under the main menu. Some Important Pages OMS: Heath Overview Navigation: > Setup -> Manage Cloud Control -> Health Overview Useful Region: Overview Region Number & Status of all targets Upload & Performance Region Loader & Network Problems OMS & Repository Management

Health OverView OMS & Repository Management

Health Overview Page OMS & Repository Management

OMS & Repository Menu To add additional OMS in existing environment. Navigation: OMS and Repository -> Add OMS This will start “Deployment Procedure Manager” OMS & Repository Management

OMS Management from Command Line OPMNCTL utility was used. However this utility does not exist anymore. This utility has been suppressed by EMCTL. EMCTL utility can be user for many operations against the OMS. OMS & Repository Management

EMCTL Examples Change Repository Password Show Repository Details > emctl config oms –change_repos_pwd [-change_in_db] [-old_pwd] [-new_pwd] [-use_sys_pwd] [-sys_pwd] Show Repository Details > emctl config oms –list_repos_details Dump all Logs > emctl dump logs List all properties from {emoms.properties} > emctl list properties –sysman_pwd “sysman pwd” –module > emctl start oms > emctl stop oms -all OMS & Repository Management

OMS Directories OMS is installed into an Middleware Home Middleware Home may contain Oracle Weblogic Server Oracle Management Agent Instance Base Directory Development Kit [ jdk ] Other Configuration Files Note:- To troubleshoot problem you need to be familiar with Directory Structure. OMS & Repository Management

Directories Structure UserProject Agent jdk module Oracle_BI1 Oracle_WI oms gc_inst logs oracle_common plugin wlserver10.3 bin sysman log em/ EMGC_OMS1 Node Manager emnodemanager Server EMGC_ADMINSERVER WebTier IH1 EMGC_OMS1 OMS & Repository Management

OMS Log & Trace Files There are 6 main log and trace files for troubleshooting. emctl.log Output of EMCTL commands emctl.msg Generated when OMS restart because of critical errors emoms.log Log file for console, created when OMS perform any action emoms.trc Contains detailed information emoms_pbs.log Contains warning for back ground modules such as event or job emoms_pbs.trc Contains more detail tracing info to troubleshoot background modules OMS & Repository Management

Oracle Management Repository Oracle Database Store Information collected by Agents. Composed of database users, tablespaces, views, indexes and other database objects. Like Agents & OMS, Repository can also be controlled from both Console and Command line utility. OMS & Repository Management

Repository Management from Console Navigation > Setup -> Manage Cloud Control -> Repository OMS & Repository Management

Repository Home Page Important things to checked: Space Available Backlog of Repository collections if Increasing Oracle Scheduler Status (if Ok) Management Service AQ status (if Ok) OMS & Repository Menu Repository Repository Synchronization {> emctl resync repos –full } Note: - Repository Synchronization can not be done from Console ? OMS & Repository Management

Repository Management from Command Line OMR can be managed neither by EMCTL nor EMCLI.. ANY GUESS.. WHY ? DBA’s Old Friend – SQL*Plus Note:- Some Important Points Stop OMS prior to stopping or restarting repository database. Make sure listener is started before the repository database is started. OMS & Repository Management

Important Area for OMR Management Data Retention Queue Management Optimizer Statistics Repository Views OMS & Repository Management

Troubleshooting EMDIAG: utility for troubleshooting across all three stacks. EMDIAG is a collection of scripts & tools designed to help troubleshoot & diagnose problems with EM. Each EM tier has a specific kit. AGTVFY to diagnose Agent problems. OMSVFY to diagnose OMS problems. REPVFY to diagnose Repository Issue. Note:- 421053.1 Master Key note for EMDIAG, explains how to download, install & use each kit. OMS & Repository Management

Troubleshooting Examples Each kit is a set of commands: to display information Setup, configuration or to dump objects Levels: varies from 0 to 9 While using each kit one should start from 0 as level parameter. Level 0 to 4 ------- functional Issue Level 6 to 8 ------- Reporting Issue Level 9 ------- Internal Diagnostics $ ./repvfy verify all –level 2 details $ ./repvfy verify all –level 9 details OMS & Repository Management

Thank You OMS & Repository Management