=== Upgrading the OMS to 13.4 === * [[https://www.martinberger.com/category/em13c-cloud-control/|Good clear instructions on patching etc. - Martin Berger]] * [[https://www.alfredokriegdba.com/tag/oem-13c/|Good doc on upgrading Enterprise Manager Cloud Control to 13.4]] * [[https://blogs.oracle.com/oem/oracle-enterprise-manager-134-is-now-available-for-download|Oracle Enterprise Manager 13.4 is now available for download - Oracle blog]] * [[https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emupg/overview-upgrading-enterprise-manager-cloud-control-13c.html|Overview of Upgrading to Enterprise Manager Cloud Control 13c Release 4]] * [[https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emupg/deleting-old-oms-home-when-central-agent-is-migrated-location-old-middleware-home.html|Removing the old homes after upgrading to 13.4]] Should the old agent be de-installed before the new one is installed? Seems to work without doing this: $/perl/bin/perl /sysman/install/AgentDeinstall.pl -agentHome ==== Create a Segment Advisor task ==== declare tname varchar2(128) := 'seg_task_1'; tname_desc varchar2(128) := 'Get shrink advice for segments in tablespace'; task_id number; object_id number; object_name varchar2(100); object_type varchar2(100); begin dbms_advisor.create_task ( 'Segment Advisor' , task_id , tname , tname_desc , null ); dbms_advisor.create_object ( tname , 'TABLESPACE' , 'USERS' , ' ' , ' ' , null , ' ' , object_id ); dbms_advisor.set_task_parameter ( tname , 'RECOMMEND_ALL' , 'TRUE' ); end; / declare tname varchar2(128) := 'seg_task_1'; begin dbms_advisor.execute_task (tname); end; / select description from dba_advisor_tasks where task_name = 'seg_task_1' / ==== Where are the logfiles in 13c? ==== * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=2112329.1|This note shows the directory structure and log file locations of an Enterprise Manager installation]] ==== agent is blocked and needs resynchronization ==== Performing emctl status agent shows Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 13.4.0.0.0 OMS Version : (unknown) Protocol Version : 12.1.0.1.0 Agent Home : /cln/exp/ora_bin1/app/oracle/agent/agent_inst Agent Log Directory : /cln/exp/ora_bin1/app/oracle/agent/agent_inst/sysman/log Agent Binaries : /cln/exp/ora_bin1/app/oracle/agent/agent_13.4.0.0.0 Core JAR Location : /cln/exp/ora_bin1/app/oracle/agent/agent_13.4.0.0.0/jlib Agent Process ID : 36176236 Parent Process ID : 46793064 Agent URL : https://hn1111.cln.be:3872/emd/main/ Local Agent URL in NAT : https://hn1111.cln.be:3872/emd/main/ Repository URL : https://hn1627.cln.be:4903/empbs/upload Started at : 2021-02-08 16:23:08 Started by user : oracle Operating System : AIX version 7.1 (ppc64) Number of Targets : 13 Last Reload : (none) Last successful upload : (none) Last attempted upload : 2021-02-08 16:23:22 Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload : 86 Size of XML files pending upload(MB) : 0.28 Available disk space on upload filesystem : 22.24% Collection Status : Collections enabled Heartbeat Status : Agent is blocked Blocked Reason : Plug-in mismatch found between agent and repository. Please perform an agent resync to unblock the agent. Mismatched plug-ins: [ The plug-in is oracle.sysman.si Last attempted heartbeat to OMS : 2021-02-08 18:29:26 Last successful heartbeat to OMS : (none) Next scheduled heartbeat to OMS : 2021-02-08 18:32:26 From the repository select * from sysman.mgmt_blocked_agents; From Enterprise Manager Cloud Control * Navigate to Setup > Manage Cloud Control > Agents * Click problematic agent * Click on drop down menu "Agent" and Choose "Resynchronization" in the menu list Click OK and the resync job will run and unblock the agent once complete. It will take a few minutes to show up as cleared in Enterprise > Summary. The same thing can be accomplished on the command-line from the oms emcli login -username=sysman -password= emcli sync emcli resyncAgent -agent="hn1111.cln.be:3872" If resync still gives issues, see [[https://support.oracle.com/epmos/faces/DocumentDisplay?parent=DOCUMENT&sourceId=1919191.1&id=1951336.1|MOS Note 1951336.1]] === Upgrading the agent on each database host === Check the ports needed with this emctl status agent Download the AgentPull.sh script curl "https://hn1627.cln.be:4903/em/install/getAgentImage" --insecure -o /tmp/AgentPull.sh Check what agents are available for download from the OMS emcli login -username=sysman emcli sync emcli get_supported_platforms or /tmp/AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=password PLATFORM=AIX -showPlatforms or curl --insecure 'https://hn1627.cln.be:4903/em/install/getAgentImage?script=show' Download the agent so it can be distributed to the db servers manually (in case curl is not available on the satellites) /tmp/AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=password PLATFORM=AIX -download_only or manually from the oms mkdir /tmp/agentinstaller emcli get_agentimage -destination=/var/tmp/agentinstaller -platform="IBM AIX on POWER Systems (64-bit)" -version="13.4.0.0.0" ==== Download the software library catalog from Metalink/MOS if using offline mode and the zip file does not save correctly ==== Hint though: From "Setup : Provisioning and Patching : Offline Patching", right click and save on the zip link instead of left clicking it! wget --user= --ask-password=on -nv https://updates.oracle.com/download/em_catalog.zip Then from "Setup : Extensibility : Self Update", choose "Check Updates" button and follow instructions.\\ When the file is downloaded and transferred onto the oms server... wget --http-user= --http-password= -O /tmp/p9348486_112000_Generic.zip https://updates.oracle.com/Orion/Download/download_patch/p9348486_112000_Generic.zip emcli login -username=sysman emcli sync emcli import_update_catalog -file=/tmp/p9348486_112000_Generic.zip -omslocal emcli logout This will now show the latest software available on the Self Update page. Now we need to download the 13.4 agent and upload it back into the software library so that it can be distributed.\\ From "Setup : Extensibility : Self Update : Agent Software", choose the required agent and click the Download button. A box will open with offline instructions. wget --http-user= --http-password= -O /tmp/p30812887_112000_Generic.zip https://updates.oracle.com/Orion/Services/download/p30812887_112000_Generic.zip?aru=23357833&patch_file=p30812887_112000_Generic.zip emcli login -username=sysman emcli sync emcli import_update -omslocal -file=/tmp/p30812887_112000_Generic.zip emcli logout A rough script to upgrade the agent ==== Create a response file ==== printf "%s" "Enter sysman password: " read LOGIN_PASSWORD printf "%s" "Enter agent registration password: " read AGENT_REGISTRATION_PASSWORD cat</tmp/em_agent.rsp LOGIN_USER=sysman LOGIN_PASSWORD=${LOGIN_PASSWORD} PLATFORM="AIX" VERSION=13.4.0.0.0 AGENT_REGISTRATION_PASSWORD=${AGENT_REGISTRATION_PASSWORD} AGENT_BASE_DIR=${ORACLE_BASE}/agent AGENT_INSTANCE_HOME=${ORACLE_BASE}/agent/agent_inst AGENT_PORT=3872 EM_UPLOAD_PORT=4903 OMS_HOST=hn1627.cln.be ORACLE_HOSTNAME=$(hostname) EOCAT ==== Source to get variables into current session ==== . /tmp/em_agent.rsp curl -sk https://${OMS_HOST}:7803/em/install/getAgentImage > /tmp/AgentPull.sh chmod 755 /tmp/AgentPull.sh ==== install the agent ==== /tmp/AgentPull.sh -ignorePrereqs RSPFILE_LOC=/tmp/em_agent.rsp ==== Post Install steps ==== $AGENT_HOME/root.sh ==== Promoting PDB failing with ORA-01422: exact fetch returns more than requested number of rows (Doc ID 2098371.1) ==== A Container database was getting discovered but failed to promote. The status stayed yellow (pending). == Cause == A duplicate entry exists in the repository for the parent of the PDB target (ie. the CDB one).
SELECT prnt.target_guid, agt.emd_url, prnt.monitoring_mode, prnt.host_name, prnt.promote_status
FROM em_targets prnt,
mgmt_assoc_instances inst,
mgmt_assoc_type_extends ext,
mgmt_targets agt
WHERE prnt.target_guid = inst.source_me_guid
AND inst.dest_me_guid = (select entity_guid from EM_MANAGEABLE_ENTITIES where entity_name='' and entity_type='oracle_pdb')
AND inst.assoc_type = ext.assoc_type
AND ext.extended_assoc_type = 'app_composite_contains'
AND agt.emd_url = prnt.emd_url
AND agt.target_type = 'oracle_emd' ;
== Solution == - Open Monitoring configuration page for the CDB target. - Remove the "Pluggable Databases" listed there. - Run an "Sync Pluggable databases" - Save the changes. ===== Some useful emcli commands ===== * Some useful emcli commands and a nice document (if only I understood German) [[https://www.doag.org/formes/pubfiles/8937684/CC132_DOAG_2017_-Verwaltung_mit-_EMCLI.pdf]] These tips and more at [[https://www.doag.org/formes/pubfiles/8937684/CC132_DOAG_2017_-Verwaltung_mit-_EMCLI.pdf|From doag.org - includes different ways of installing agents]] Login to the cli and sync the client emcli login -username=sysman [-password=] emcli sync === List all the agents === emcli get_targets -targets="oracle_emd" Status Status Target Type Target Name ID 1 Up oracle_emd hn492.cln.be:3872 1 Up oracle_emd hn1111.cln.be:3872 1 Up oracle_emd hn481.cln.be:3872 1 Up oracle_emd hn991.cln.be:3872 1 Up oracle_emd hn511.cln.be:3872 1 Up oracle_emd hn990.cln.be:3872 1 Up oracle_emd hn1627.cln.be:3872 1 Up oracle_emd hn482.cln.be:3872 4 Agent Unreachab oracle_emd hn5306.cln.be:3872 le 1 Up oracle_emd hn5405.cln.be:3872 1 Up oracle_emd hn491.cln.be:3872 1 Up oracle_emd hn5410.cln.be:3872 1 Up oracle_emd hn5308.cln.be:3872 1 Up oracle_emd hn5407.cln.be:3872 1 Up oracle_emd hn512.cln.be:3872 === List all databases === emcli get_targets -targets="oracle_database" Status Status Target Type Target Name ID 1 Up oracle_database bkstt 1 Up oracle_database credot 1 Up oracle_database rpae 1 Up oracle_database rpad 1 Up oracle_database appli 1 Up oracle_database credoa 1 Up oracle_database credoe 1 Up oracle_database reportt 1 Up oracle_database qfp 0 Down oracle_database dwhm 1 Up oracle_database itsm 1 Up oracle_database xbrla 1 Up oracle_database dwht 1 Up oracle_database qfa 1 Up oracle_database rpaa 1 Up oracle_database org 1 Up oracle_database dwha 1 Up oracle_database dwhd 4 Agent Unreachab oracle_database crelsast le 1 Up oracle_database xbrl 1 Up oracle_database crelsasp 1 Up oracle_database atm 1 Up oracle_database ite 1 Up oracle_database amlt === Add a database target to Cloud Control via emcli === * [[https://dbaclass.com/article/emcli-command-oem-12c/|Add target database to OEM cloud]] emcli add_target -name="ORACLE_SID" -type="oracle_database" -host="$oemhost" -credentials="UserName:dbsnmp;password:dbsnmp_password;Role:Normal" -properties="SID:ORACLE_SID;Port:1521;OracleHome:;MachineName:\" e.g. emcli add_target -name="adsdd1" -type="oracle_database" -host="hn1627.cln.be" -credentials="UserName:dbsnmp;password:dbsnmp_password;Role:Normal" -properties="SID:adsdd1;Port:3528;OracleHome:/cln/dev/ora_bin/app/oracle/product/19.13/dbhome_1;MachineName:hn1627.cln.be" === Get list of plugins installed on OMS server === ./emcli list_plugins_on_server OMS name is dbadev64.dba.com.sa:4889_Management_Service Plug-in Name Plugin-id Version [revision] Oracle Audit Vault oracle.em.soav 12.1.0.4.0 Oracle GoldenGate oracle.fmw.gg 13.1.1.0.0 Oracle Cloud Framework oracle.sysman.cfw 13.2.1.0.0 Oracle Database oracle.sysman.db 13.2.1.0.0 Oracle Fusion Middleware oracle.sysman.emas 13.2.1.0.0 === Get list of plugins installed on a Management Agent === ./emcli list_plugins_on_agent -agent_names="172.25.224.2:3872" The Agent URL is https://172.20.204.2:3872/emd/main/ - Plug-in Name Plugin-id Version [revision] Oracle Home oracle.sysman.oh 13.2.0.0.0 Oracle Database oracle.sysman.db 13.2.1.0.0 === Create blackout === emcli create_blackout -name="DBACLASS_MULTDB_BLACKOUT" -reason="psu patchcing" -add_targets="DBATEST1:oracle_database;DBATEST2:oracle_database" -schedule="duration:30" === Create blackout for a host(target agent) and all this services/database runnin on that server === ./emcli create_blackout -name="DBACLASS_HOST_BLACKOUT" -reason="OS PATCHING OUTAGE" -add_targets="localhost.dbaclass.com:host" -schedule="duration:60" -propagate_targets === Get status of a blackout === ./emcli get_blackout_details -name="db_blackout_phase1" === Stop a blackout === ./emcli stop_blackout -name="db_blackout_phase1" ./emcli get_blackout_details -name="db_blackout_phase1" === Delete a blackout === ./emcli delete_blackout -name="db_blackout_phase1" ./emcli get_blackout_details -name="db_blackout_phase1" === Stop and start an agent from OMS server === ./emcli stop_agent -agent_name="dbahost-local:3872" -host_username=oracle ./emcli start_agent -agent_name="dbahost-local:3872" -host_username=oracle === Create a Super Administator account === emcli create_user -name="stuart" -password="password" -privilege="SUPER_USER" === Create an Administrator account with some privileges === emcli create_user -name="someonelse" -password="password" -roles="public;EM_USER" -privilege="OPERATOR_ANY_TARGET" -privilege="CONNECT_ANY_VIEW_TARGET" -privilege="CREATE_JOB" === Show all the Administrator accounts === emcli list -resource="Administrators" -columns="USER_NAME,USER_DESCRIPTION" === Show all active sessions === emcli list_active_sessions -details -table === Change a database user password === emcli update_db_password -target_name=prod1 -user_name=stuart -old_password= -new_password= -retype_new_password= -change_at_target=yes === Unlock a database user account === emcli update_db_account_status -target_name=prod1 -user_name=DBSNMP -action=UNLOCK -connect_as="DBNamedCreds:NC_SYS" or emcli update_db_account_status -target_name=dwhm -user_name=DBSNMP -action=UNLOCK -connect_as="DBUserName:sys;DBPassword:sys;DBRole:sysdba" === List groups (including dynamic ones) === emcli get_groups Target Name Target Type ACC group All Instances group DEV_TST_BLD group EXP group KAP group === Show group members === emcli get_group_members -name=KAP Target Name Target Type EBSO oracle_database acc_perf oracle_database adso oracle_database amlk oracle_database bilk oracle_database clnk oracle_database crelsask oracle_database dwh_perf oracle_database loank oracle_database monprodk oracle_database orgperf oracle_database portalk oracle_database reportk oracle_database === Create a job by copying an existing one === For example, there is now a library job called "Change the Password for the Database Monitoring User" (changes the dbsnmp password on one or many databases!). * Run it once so that there is a job in the Activity page. * Find the internal name of the job by describing it. emcli describe_job -name="CHPASSDBSNMP">/tmp/CHPASSDBSNMP.txt cat /tmp/CHPASSDBSNMP.txt name=CHPASSDBSNMP type=dbChangeDBPassword owner=SYSMAN target_list=upg:oracle_database variable.new_password=****** variable.retype_new_password=****** schedule.frequency=IMMEDIATE * This text file can now be edited and used to create a job. Leaving out the variable names for password allows Cloud Control to generate a random password. vi /tmp/CHPASSDBSNMPALL.txt name=CHPASSDBSNMPALL.1 type=dbChangeDBPassword owner=SYSMAN target_list="All Instances":group schedule.frequency=IMMEDIATE emcli create_job -input_file=property_file:/tmp/CHPASSDBSNMPALL.txt emcli get_jobs -name="CHPASSDBSNM%" or this one schedules a regular jobs to do the password change # Current status of the job is ACTIVE. name=CHANGE DBSNMP PASSWORD EVERYWHERE.1 type=dbChangeDBPassword owner=SYSMAN target_list=All Instances:composite schedule.frequency=MONTHLY schedule.startTime=2022-01-11 11:11:11 schedule.days=11 schedule.timezone.region=Target/Local notification.match=EITHER properties.max_concurrent_executions=10 # download emcli to add targets wget -P /tmp/emcli --quiet --no-check-certificate https://${OMS_HOST}:7803/em/public_lib_download/emcli/kit/emclikit.jar [[ $? -ne 0 ]] && echo "ERROR: wget is not working, check the version works with ssl" && exit 1 export JAVA_HOME="${OMS_HOME}/oracle_common/jdk/jre/" export PATH="${JAVA_HOME}:${PATH}" java -jar /tmp/emcli/emclikit.jar -install_dir=/tmp/emcli # configure emcli /tmp/emcli/emcli setup -url=https://${OMS_HOST}:7803/em -username="${LOGIN_USER}" -password="${LOGIN_PASSWORD}" -dir="/tmp/emcli/conf" -trustall -noregister -autologin # generate target addition statements echo "add_target -name=\"linuxlab01.local_listener\" -type=\"oracle_listener\" -host=\"linuxlab01.local\" -properties=\"LsnrName:LISTENER;ListenerOraDir:/app/oracle/product/12.1.0/network/admin;Port:1521;OracleHome:/app/oracle/product/12.1.0;Machine:linuxlab01.local;\"" >> "/tmp/emcli/emclicommands.txt" ORAENV_ASK=NO RUNNING_SIDS="$(ps -efo args|grep '[p]mon'|awk -F'ora_pmon_' '! /\+ASM|^$/ {print $NF}'|sort -fu)" for i in ${RUNNING_SIDS}; do export ORACLE_SID="${i}" . oraenv >/dev/null 2>&1 [[ $? -ne 0 ]] && echo "WARNING: CAnnot set environment for ${i}, skipping" && continue echo "add_target -name=\"${i}\" -type=\"oracle_database\" -host=\"${ORACLE_HOSTNAME}\" -credentials=\"UserName:dbsnmp;password:dbsnmp;Role:Normal\" -properties=\"SID:${i};Port:${LISTENER_PORT};OracleHome:${ORACLE_HOME};MachineName:${ORACLE_HOSTNAME};\"" >> "/tmp/emcli/emclicommands.txt" done # load up the targets /tmp/emcli/emcli login -username="${LOGIN_USER}" -password="${LOGIN_PASSWORD}" /tmp/emcli/emcli sync tmp/emcli/emcli argfile /tmp/emcli/emclicommands.txt # remove the emcli rm -rf /tmp/emcli ==== Get the agent to autodiscover and upload the basic targets like the agent itself and the host ==== emctl config agent addinternaltargets ==== Run autodiscovery on a target host from OMS server ==== First, create a job properties file ./emcli describe_job_type -job_type=RunAutoDiscovery > inputs.prop # Description: (Optional) The user specified name of the job name=AUTO_DISC # Description: (Optional) The job type for this job type=RunAutoDiscovery # Description: (Optional) The user specified description of the job description= # Description: The job owner. The job owner is the user who creates the job. # Default: the logged in user # The job owner information displayed here is for documentation only and user is not expected to change it. owner= # Description: (Optional) The kind of job # Legal Values: active, library kind= # Fill in the target list before submitting. # For Example: # target_list=MyTarget:cluster target_list=172.20.294.137:host # Description: The type of targets to use for this job target ype=host # Description: (Optional) Notify the job owner when a selected state occurs # Allowed Values: SCHEDULED, RUNNING, ACTION_REQUIRED, SUSPENDED, SUCCEEDED, PROBLEMS notification= # Description: (Optional) The run time properties for the job properties=
Finally, create (run)a job ./emcli create_job -input_file=property_file:inputs.prop Check the job status ./emcli get_jobs -name=AUTO_DISC === General info on Oracle Enterprise Manager Cloud/Grid Control === * [[http://www.oracle.com/technetwork/oem/grid-control/twp-security-best-practices-133704.pdf|Grid security best practices]] * [[https://docs.oracle.com/cd/cloud-control-13.3/EMVWS/EMVWS.pdf|OEM Cloud Control 13c Management Views]] === Additional references: === * Note 951076.1: How to Troubleshoot Communication From a Grid Agent to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control? * Note 1089443.1: How to Troubleshoot Communication From the Grid Console (UI) Machine to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control? * Note 1089693.1: How to Troubleshoot Communication From the Oracle Management Service (OMS) to the Grid Control Repository Database in 10g Enterprise Manager Grid Control? The Enterprise Manager Grid Control includes the following Components: * OMS - Management Service - This is responsible for communicating with the user via a GUI and communicating with the OMR * OMR - Management Repository - This is a collection of tables owned by the sysman schema that stores all the data collected by the OMA * OMA - Management Agent - This is a perl program that runs on the database host (one for all databases on the host) and uploads data to the OMS for storage in the OMR * [[https://dbaclass.com/upgrade-oem-cloud-control-from-12c-to-13c/|Upgrade OEM 12C To 13C - dbaclass.com]] * [[https://support.oracle.com/epmos/faces/CertifyResults?searchCtx=st%255EANY%257Cpa%255Epi%255E931_Enterprise%2BManager%2BBase%2BPlatform%2B-%2BOMS%257Evi%255E775884%257Epli%255E20_Linux%2Bx86%257Eplvi%255E23258%257Epln%255ELinux%2Bx86%2BRed%2BHat%2BEnterprise%2BLinux%2B6%257E%257C&_afrWindowMode=0&_adf.ctrl-state=egqooixsw_4|Oracle Enterprise Management (OEM) Compatibility Matrix]] ==== A useful set of emcli commands ==== * [[https://laurent-leturgez.com/2012/06/11/useful-emcli-commands-in-em-cloud-control-12c/|laurent-leturgez.com]] * [[https://www.mydbaworld.com/install-em13c-agent-add-targets-using-command-line/|Install EM13c agent and add targets using the Linux command line only]] * [[http://facedba.blogspot.com/2018/02/named-credential-management-in-oem-13c.html|Named Credential Management in OEM 13c using EM CLI]] emcli list -resource="Administrators" emcli list -resource="Administrators" -help emcli list -resource="Administrators" -columns="USER_NAME,REPOS_ACCOUNT_STATUS" emcli list -resource=Administrators -columns="USER_NAME,USER_TYPE:20,COST_CENTER,CONTACT,DEPARTMENT:30" and same thing in a different way emcli list -resource=Administrators -columns="USER_NAME,USER_TYPE,COST_CENTER,CONTACT,DEPARTMENT" -colsize="USER_TYPE:20,DEPARTMENT:30" Some resources need a bind variable to be passed to get output. A bind variable is created in the resource and then referenced in the command. For example, this command will list all the default preferred credentials for target type oracle_database. emcli list -resource="PreferredCredentialsDefault" -bind="TargetType='oracle_database'" -colsize="SetName:15,TargetType:15" emcli get_targets -targets='%database%' emcli list -resource="Targets" -columns="TARGET_TYPE,TARGET_NAME" -colsize="TARGET_TYPE:25,TARGET_NAME:60" -search="TARGET_TYPE='oracle_database'" emcli list -resource="Targets" -columns="TARGET_TYPE,TARGET_NAME" -colsize="TARGET_TYPE:25,TARGET_NAME:60" -search="TARGET_TYPE='oracle_database'" -search="TYPE_QUALIFIER3='DB'" emcli list -resource=NamedCredentials emcli list -resource=NamedCredentials | awk '{ print $2 $3}' emcli list -resource=NamedCredentials -columns="CredName,CredOwner" emcli list -resource=NamedCredentials -format=name:csv > '/path/filename.csv' Create Named Credentials with the scope of one target: To be able to create a named credential, you need to know the target name (unless you create a global credential), target type and credential type associated with the target type. Let's say I want to create a named credential for my database named "TESTDB". First I need to login to our EM12c server, and list targets named "TESTDB": ./emcli create_named_credential -auth_target_type=oracle_database \ -cred_scope=Instance -target_type=oracle_database \ -target_name=TESTDB -cred_type=DBCreds -cred_name=NC_INST_SYS_TESTDB \ -attributes="DBUserName:SYS;DBPassword:Hell0W0rld;DBRole:sysdba" -test Create GLOBAL credentials: (note -test becomes -test_target_name) emcli create_named_credential -auth_target_type=oracle_database \ -cred_scope=GLOBAL -target_type=oracle_database \ -target_name=TESTDB -cred_type=DBCreds -cred_name=NC_GL_TESTDB_SYS \ -attributes="DBUserName:SYS;DBPassword:Hell0W0rld;DBRole:sysdba" -test_target_name=Global ==== List all credentials stored in the repository ==== Connect to th repository database as sysdba and run this query set lines 1000 pages 100 col cred_owner for a10 col cred_name for a60 col cred_type_name for a20 col target_type for a20 col user_name for a10 select cred_owner , cred_name , cred_type_name , target_type , user_name , decode (cred_scope, 1, 'Global', 2, 'Target') "cred_scope" , to_char(trunc(cred_created_date),'DD-Mon-RRRR') "created_on" , to_char(trunc(cred_updated_date),'DD-Mon-RRRR') "updated_on" from sysman.em_nc_creds where cred_owner != '' order BY 1,2,3; ==== Startup the Management server, central agent and the repository in the correct order ==== * [[https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.3.1/emadm/starting-and-stopping-oracle-enterprise-manager-13c-cloud-control.html#GUID-E866CE88-ABF0-4DB9-AB0D-25DD453BACE5|Starting and Stopping Oracle Enterprise Manager 13c Cloud Control]] The Repository database (and listener!) have to be started before the Middleware components can be started. === Change sysman password === The sysman password needs to be changed with emctl. Changing it from the database is not good enough.\\ Setup the OMS environment (default is $ORACLE_BASE/middleware). 'stop oms -all' is not done before changing the password as the Admin Server needs to be up to make the change. cd $ORACLE_HOME/bin emctl stop oms emctl stop oms -bip_only emctl extended oms jvmd stop -all emctl config oms -change_repos_pwd -old_pwd oldsysmanpassword -new_pwd newsysmanpassword emctl stop oms -all emctl start oms ==== Change dbsnmp password ==== Finally,found a way of doing this without needing to know the old password! * [[https://blogs.oracle.com/observability/post/non-dmsnmp-monitoring-credentials|Thank you Desiree :-)]] emcli set_monitoring_credential -target_type=oracle_database -target_name="upg" -set_name="DBCredsMonitoring" -cred_type="DBCreds" -attributes="DBUserName:dbsnmp;DBPassword:password;DBRole:Normal" Simple when you know how! But the documentation is quite poor in terms of examples and which verb to use for which situation. Also in the same article... Adding a target database using EM CLI When adding a new database, explicitly or through discovery, you can specify non-DBSNMP monitoring credentials for the database: === add_target === emcli add_target -name="database" -type="oracle_database" -host="myhost.example.com" -credentials="UserName:monuser;password:password1;Role:Normal" -properties="SID:semcli;Port:15091;OracleHome:/oracle; MachineName:smpamp.example.com" === discover_db === emcli discover_db -input_file=db_discovery_file:db_discovery_file.txt The example file, db_discover_file.txt, may have contents like below: host_name=myhost.example.com db_creds=db1:monuser,password1,NORMAL target_type=oracle_database * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1592390.1|EM 13c: How to Update DBSNMP Password and Change Monitoring Credentials of an Enterprise Manager Cloud Control Target Database Using emcli Command Line (Doc ID 1592390.1)]] 1) Single Database Instance How to Use update_db_password verb emcli login -username=sysman -password= ./emcli get_targets -target="oracle_database" emcli update_db_password -target_name="" -target_type="oracle_database" -user_name="dbsnmp" -change_at_target=yes -old_password="" -new_password="" -retype_new_password="" => Replace each value in <> in command line with the appropriate values. Note that if the database has been independently updated for the DBSNMP password, use of this verb must accommodate this fact. In such a case, the "-change_at_target" option is no longer needed. Documentation on update_db_password verb: For EM 13c: Enterprise Manager Command Line Interface\\ Release 13.2\\ Chapter 5 Verb Reference\\ 5.4 EM CLI Verbs\\ [[https://docs.oracle.com/cd/E24628_01/em.121/e17786/cli_verb_ref.htm#EMCLI1505|5.4.714 update_db_password]] === Login to EM on OMS === emcli login -username=sysman [[-password=]] === logout of EM on OMS === emcli logout === Sync emcli with OMS === emcli sync === List promoted targets === emcli get_targets === Delete a specific target === emcli delete_target -name="devtools" -type="oracle_database" Target "devtools:oracle_database" deleted successfully === Find all agents === emcli get_targets -format=name:csv | grep "oracle_emd" === Delete an agent and ALL its targets === emcli delete_target -name="hn5407.cln.be:3872" -type="oracle_emd" -delete_monitored_targets === Follow a plugin deployment (on the OMS / on an agent) === Oracle Database plugin emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db Oracle Fusion Middleware plugin emcli get_plugin_deployment_status -plugin_id=oracle.sysman.emas My Oracle Support plugin emcli get_plugin_deployment_status -plugin_id=oracle.sysman.mos === Import an update (for example: a plugin update) into the software library === emcli import_update -file="p14018177_112000_Generic.zip" -omslocal === Deploy a plugin on the OMS === emcli deploy_plugin_on_server -plugin=oracle.sysman.db -sys_password=XXXXX === Deploy a plugin on EM agent(s) === emcli deploy_plugin_on_agent -plugin="oracle.sysman.db" -agent_names="xxxdb-prod1.mydomain.com:3872;xxxdb-prod2.mydomain.com:3872" === List available agents in the library === emcli get_supported_platforms === Download an agent from the library (used for agentDeploy.sh script method) === emcli get_agentimage -destination=/home/oracle -platform="Microsoft Windows x64 (64-bit)" -version="12.1.0.1.0" === Set monitoring credentials for a specific target (example given for an Oracle database instance) === emcli set_credential -target_type=oracle_database -target_name="prod1" -credential_set=DBCredsMonitoring -user=sysman -column="Role:SYSDBA;UserName:sys;password:XXXXX" -monitoring ==== Uninstall the agent oracle home that registered with inventory ==== If you are getting "agent home is already registered with the inventory", it means there is an entry for this home in OraInventory.xml already and it will need removing before installing to the same directory.\\ Agent Homes are stored in /etc/oragchomelist (Linux, AIX) and /var/opt/oracle/oragchomelist (Solaris).\\ Removing the entry from oragchomelist makes no difference.\\ It's in the Oracle Inventory file – its location can be found in /etc/oraInst.loc (Linux, AIX) and /var/opt/oracle/oraInst.loc (Solaris). ORACLE_HOME="/oracle/cloud/core/12.1.0.1.0" $ORACLE_HOME/oui/bin/runInstaller -silent -detachHome ==== Get rid of an agent that won't go away in the Grid screens ==== This has the effect of removing everything related to the host in question!\\ On host where agent is running... emctl stop agent On Grid server... sqlplus / as sysdba exec mgmt_admin.cleanup_agent('>'); exit On host where agent is running... emctl start agent ==== Get rid of targets that won't go away in the Grid screens: ==== exec mgmt_admin.delete_target('target_name','target_type') See mgmt_targets table in sysman schema for list of known targets. ==== Re-discover targets ==== cd $ORACLE_HOME/bin ./agentca -d ==== Agent Unreachable ==== * Is the agent running? From SQL*Plus: select username,program from v$session where LOWER (program) like 'emagent%'; If no rows selected, the agent is not running. or From Unix: . oraenv agent11g (how to see [[which databases are running on the machine]]) emctl status agent ps -ef|grep [[a]]gent There should be a handful. If only a few, kill them and restart the agent. emctl start agent //Check the agent log in $ORACLE_HOME/sysman/log// ==== Start a host level blackout from command line (blackout has to already have been created via OEM) ==== This could be included in a shell script before patching, for example... emctl start blackout server_maint -nodeLevel ==== Start a host level blackout for a certain duration ==== This starts a blackout from now until now + 8 hours emctl start blackout server_maint -nodeLevel -d 08:00 ==== Stop a blackout from command line ==== When patching is done?... emctl stop blackout server_maint ==== [[Backend WLS or EM application seems to be down]] ==== ==== To list targets known to an agent: ==== emctl config agent listtargets It looks at the file $AGENT_HOME/sysman/emd/targets.xml Manually add targets by editing this file and running: emctl config agent addtargets $AGENT_HOME/sysman/emd/targets.xml Check state and upload directories under $AGENT_HOME/sysman/emd for .err files ==== Disable BI Publisher ==== $OMS_HOME/bin/emctl config oms -disable_bip Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Stopping BI Publisher Server... BI Publisher Server Successfully Stopped BI Publisher Server is Down BI Publisher has been disabled on this host and will not be started with the 'emctl start oms' or 'emctl start oms -bip_only' commands. Overall result of operations: SUCCESS ==== Alter Page Timeout ==== To avoid the message "The page has expired. Click OK to continue". cd $OMS_HOME/bin ./emctl set property -name oracle.sysman.eml.maxInactiveTime -value -1 -sysman_pwd sysman_password ./emctl stop oms ./emctl start oms Disable the feature altogether with '-1' value (it's null by default), or enter number of minutes. ==== Change Repository listener port ==== If the Oracle listener port is changed, the OMS will need to be reconfigured to use the new port. emctl stop oms -all Start the administration server emctl start oms -admin_only Update the OMS with the changed listener port emctl config oms –store_repos_details -repos_conndesc -repos_user sysman Stop and restart te OMS emctl stop oms -all emctl start oms ==== Create 2048 bit SSL Certificate Authority ==== * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1611578.1|Regenerating OEM 12c/13c SSL certificates with Higher Keystrength and Signature Algorithm (Doc ID 1611578.1)]] emctl secure createca -key_strength 2048 emctl stop oms emctl start oms emcli login -username=sysman emcli sync emcli get_ca_info -details Then secure the agents so thay use the new CA [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1390222.1]]\\ Secure the OMS after securing all the Agents emctl secure oms -console [-protocol TLSv1] Restart everything emctl stop oms -all emctl start oms ==== Change sysman password ==== This works until 11.1. For a more complete guide see metalink note [[https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=390374477625884&parent=DOCUMENT&sourceId=1111894.1&id=270516.1&_afrWindowMode=0&_adf.ctrl-state=n4wa0tnro_294|270516.1]] or note [[https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=n4wa0tnro_99&_afrLoop=390817620003957|259379.1]] //For the DB Control Release 11.2 and higher, you need to set the environment variable ORACLE_UNQNAME to the value of the DB_UNIQUE_NAME database parameter.// Changing this password is easy as long as it is done correctly. It is a three step process.\\ Step 1. Change the password in the traditional manner. SQL> alter user sysman identified by &new_password account unlock; Step 2. Change the password in the emoms.properties file. vi ${AGENT_HOME}/sysman/config/emoms.properties Change the following 2 lines by entering the clear text password where the encrypted password is, and set True to False oracle.sysman.eml.mntr.emdRepPwd=\\ oracle.sysman.eml.mntr.emdRepPwdEncrypted=False\\ Step 3. Restart the agent. Picks up the new password (and encrypts it) emctl restart agent ==== Change dbsnmp password (This is 10g. There's a better way now! Search again ==== Changing this password is easy as long as it is done correctly. It is a three step process.\\ === Change the password in the traditional manner === SQL> alter user dbsnmp identified by &new_password account unlock; === Change the password in the targets.xml file === vi ${AGENT_HOME}/sysman/emd/targets.xml Change the following line by entering the clear text password where the encrypted password is, and set True to False === Restart the agent. Picks up the new password (and encrypts it) === emctl restart agent ==== Quick Checklist ==== borrowed from [[http://oraxprt.com/2013/08/how-to-troubleshoot-communication-from-the-oracle-management-service-oms-to-a-grid-agent-in-10g-enterprise-manager-grid-control.html|oraxprt.com]] Verify that the Agent on the target machine is up and running using: cd /bin emctl status agent The command should return output such as: emctl status agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. ————————————————————— Agent Version : 10.2.0.5.0 OMS Version : 10.2.0.5.0 Protocol Version : 10.2.0.5.0 Agent Home : /home/oracle/OracleHomes/agent10g Agent binaries : /home/oracle/OracleHomes/agent10g Agent Process ID : 24465 Parent Process ID : 24449 Agent URL : https://agentmachine.domain:1830/emd/main/ Repository URL : https://omsmachine.domain:1159/em/upload Started at : 2010-04-22 15:35:39 Started by user : oracle …. ————————————————————— which indicates that the Agent has started up fine. Also review the /sysman/log/emagent.nohup to ensure that the Agent is not re-starting frequently, which can affect the OMS to Agent communication. Refer to Note 548928.1: Enterprise Manager Grid Control Agent 10g, Process Control (Start, Stop & Status) Troubleshooting Guide Verify that the Agent's URL, as seen in the Grid Console -> Setup -> Agent name page is the same as the value configured for the EMD_URL in the /sysman/config/emd.properties file. Refer Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent? 1. OMS / Agent Component level issues If the Agent machine is configured with DHCP and/or the IP address of the machine has recently changed, the OMS will not be able to communicate with the Agent. Refer Note 605009.1: Problem: OMS Cannot Communicate with Agent if IP Address of the Grid Agent Machine is Changed If there is a rogue emagent process on the target machine, then the OMS log/trace files could show communication errors. Refer Note 733879.1: Communication: OMS Log/Trace Files Show 'ERROR eml.OMSHandshake processFailure' for Agent Already Removed from Grid Console If the Agent is not capable of accepting incoming connection requests from the OMS, then the communication will fail. Refer Note 550452.1: Communication: OMS to Agent Communication Fails with 'IOException in sending Request :: Broken pipe' Verify if there are multiple Agents installed / discovered from this machine. Refer to Note 435728.1: Communication: OMS to Agent Communication Fails with "Connection refused" if Multiple Agent Targets are Discovered 2. Hostname/IP Address Resolution Issues If the OMS and Agent Components are located in separate machines, then the hostname/IP address resolution should work correctly from the OMS to the Agent machine. Refer Note 763844.1: How to Verify the Hostname/IP Address Resolution Between the 10g Enterprise Manager Grid Control Components? If the OMS is unable to resolve the hostname / IP address of the Agent machine, the /sysman/log/emoms.trc will show errors such as below, when trying to access the Agent Homepage in the Grid Console: 2010-04-26 12:01:51,405 [[EMUI_12_01_26_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdUploadData.1732 – IOException in sending Request :: No route to host To verify the Hostname / IP Address resolution from OMS to Agent machine, follow below steps: Collect the following details on the Agent machine: Hostname and the corresponding IP Address on which the Agent is configured: cd /bin emctl status agent Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0. Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. ————————————————————— Agent Version : 10.2.0.5.0 OMS Version : 10.2.0.5.0 Protocol Version : 10.2.0.5.0 Agent Home : /home/oracle/OracleHomes/agent10g Agent binaries : /home/oracle/OracleHomes/agent10g Agent Process ID : 24465 Parent Process ID : 24449 Agent URL : https://agentmachine.domain:1830/emd/main/ Repository URL : https://omsmachine.domain:1159/em/upload Started at : 2010-04-22 15:35:39 Started by user : oracle The hostname is the one seen in the 'Agent URL' field. Obtain the IP address for this hostname using: ping Output of these commands: ping ping ping nslookup nslookup nslookup Collect the following details from the OMS machine: ping ping ping nslookup nslookup nslookup Compare the output of the above commands on OMS and Agent machines – the outputs should match. If there is a difference or an error, please enlist the help of your System / Network Administrator to correct the configuration in the hosts file or the DNS. Note: 1. If all the above commands work fine but the OMS still fails to communicate with the Agent, then stop and restart the OMS once to reset the TCP caching /opmn/bin opmnctl stopall /opmn/bin> opmnctl startall 2. If the Agent machine has multiple NIC cards / IP addresses, the Agent can be bound to a particular hostname / IP address combination using steps in: Note 390444.1: How to: Tell the agent to listen to only one specific NIC Network Interface Card? If the hostname / resolution works fine from the OMS to Agent but the communication still fails, then check for the presence of Firewall or Proxy Server in the setup using the steps below. 3. Firewall Setup / Proxy Server Issues For details about configuring the Firewall and using the Proxy Server for the EM components, refer Note 1088393.1: How to Verify the Communication Between the 10g Enterprise Manager Grid Control Components via Firewall/Proxy? If the Agent port is blocked, then the /sysman/log/emoms.trc will show: 2008-12-01 11:21:25,535[[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdTargetsList.1767 – CommException: Unable to get list of targets from emd-getEmdTargetsList() 2008-12-01 11:21:25,541 [[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdTargetsList.1769 - Connection timed out oracle.sysman.emSDK.emd.comm.CommException: Connection timed out The following error is displayed when trying to look at the Targets -> Agent Host -> Performance page: An error has occurred! Unable to obtain data for target solaris.oracle.com. The target may be down. Switching to the last 24 hrs view Incorrect Proxy server configuration at the OMS side, can cause problems described in Note 395717.1: Communication: OMS to Agent Communication Fails With 'Cannot Establish Proxy Connection' Due to Proxy-Related Settings To verify the communication between OMS to Agent machine, when Firewall / Proxy server is in use: Identify the Agent port and URL using the steps in Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent? Test the connectivity to the Agent URL from the OMS machine, using one of the following methods: Open a web-browser on the OMS machine and try to access these URL's: http://agentmachine.domain:agentport/em/upload OR https://agentmachine.domain:agentport/em/upload The URL must return an output similar to: EMAgent10.1.0.2.0 Congratulations, EMAgent is working! Use telnet telnet agentmachine.domain Sample output: telnet agentmachine.domain 3872 Trying 20.20.20.20… Connected to agentmachine.domain. Escape character is '^]]'. If the access to the port is blocked due to a firewall, then the above command will fail with: telnet agentmachine.domain 3872 Trying 20.20.20.20… telnet: connect to address 20.20.20.20: Connection refused Use wget wget OR wget –no-check-certificate If any of the above commands fail, please contact your Network Administrator to determine if there is a Firewall / Proxy Server in use and check the configuration. ==== Uninstall OEM Cloud Control managements agent ==== * [[https://docs.oracle.com/cd/E24628_01/install.121/e24089/deinstall_agent.htm#EMADV351|Use command line to uninstall unwanted agents from database target servers]] /u01/app/Oracle/core/12.1.0.5.0/perl/bin/perl /u01/app/Oracle/core/12.1.0.5.0/sysman/install/AgentDeinstall.pl -agentHome /u01/app/Oracle/core/12.1.0.5.0/ -skipRemoval If things were installed correctly originally, the deinstall should be as easy as $AGENT_HOME/perl/bin/perl $AGENT_HOME/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME and will produce something like this Agent Oracle Home: /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0 NOTE: The agent base directory: /cln/tst/ora_bin1/app/oracle/product/agent12c will be removed after successful deinstallation of agent home. DetachHome Command executed:/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0 -waitForCompletion -invPtrLoc /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 62464 MB Passed The inventory pointer is located at /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc 'DetachHome' was successful. Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 62464 MB Passed The inventory pointer is located at /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc The Oracle home '/cln/tst/ora_bin1/app/oracle/product/agent12c/sbin' could not be updated as it does not exist. Deinstall Command executed:/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0}" -waitForCompletion -removeAllFiles -invPtrLoc /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 62464 MB Passed Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-11-28_04-47-46PM. Please wait ...Oracle Universal Installer, Version 11.1.0.11.0 Production Copyright (C) 1999, 2013, Oracle. All rights reserved. Starting deinstall Deinstall in progress (Thursday, November 28, 2019 4:47:54 PM CET) Configuration assistant "Agent Deinstall Assistant" succeeded ............................................................... 100% Done. Deinstall successful End of install phases.(Thursday, November 28, 2019 4:49:20 PM CET) End of deinstallations Please check '/cln/tst/ora_bin1/app/oracle/oraInventory/logs/silentInstall2019-11-28_04-47-46PM.log' for more details. Do you want us to delete the old oracle home [yes/no] :yes NOTE: The targets monitored by this Management Agent will not be deleted in the Enterprise Manager Repository by this deinstall script. Make sure to delete the targets manually from the Cloud Control Console for a successful deinstallation. == References == NOTE:235290.1 – Understanding the Enterprise Manager Management Agent 10g 'emd.properties' File NOTE:358953.1 – What ports are used in communication between the Grid Control OMS and a Management Agent? NOTE:471842.1 – Understanding Proxy Settings in Enterprise Manager Grid Control ==== Some useful Metalink Master Documents related to Grid Control ==== * Master Index for Managing Oracle Database and Listener with Grid Control [[ID 1304021.1]] * Master Note for 10g Grid Control Agent Process Control (Start, Stop & Status) & Configuration [[ID 1082009.1]] * How to Run the RDA against a Grid Control Installation [[ID 1057051.1]] * How to Run the RDA against a Grid Control Installation Release 11g [[ID 1190193.1]] * Grid Control Target Maintenance: Steps to Diagnose Issues Related to "Agent Unreachable" Status [[ID 271126.1]] * Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues [[ID 1086343.1]] * Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control [[ID 1202453.1]] * Receiving agent unreachable notification emails very often after 10.2.0.4 agent upgrade [[ID 752296.1]] * Healthcheck Metric failing for a 10.2.0.4 Target Database with 10.2.0.4 Agent [[ID 602633.1]]