dbawiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
=== Upgrading the OMS to 13.4 ===
  * [[https://www.martinberger.com/category/em13c-cloud-control/|Good clear instructions on patching etc. - Martin Berger]]
  * [[https://www.alfredokriegdba.com/tag/oem-13c/|Good doc on upgrading Enterprise Manager Cloud Control to 13.4]]
  * [[https://blogs.oracle.com/oem/oracle-enterprise-manager-134-is-now-available-for-download|Oracle Enterprise Manager 13.4 is now available for download - Oracle blog]]
  * [[https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emupg/overview-upgrading-enterprise-manager-cloud-control-13c.html|Overview of Upgrading to Enterprise Manager Cloud Control 13c Release 4]]
  * [[https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emupg/deleting-old-oms-home-when-central-agent-is-migrated-location-old-middleware-home.html|Removing the old homes after upgrading to 13.4]]

Should the old agent be de-installed before the new one is installed? Seems to work without doing this:
<code>
$<AGENT_HOME>/perl/bin/perl <AGENT_HOME>/sysman/install/AgentDeinstall.pl -agentHome <AGENT_HOME>
</code>
==== Create a Segment Advisor task ====
<code>
declare
    tname varchar2(128) := 'seg_task_1';
    tname_desc varchar2(128) := 'Get shrink advice for segments in tablespace';
task_id number;
    object_id number;
    object_name varchar2(100);
    object_type varchar2(100);
begin
    dbms_advisor.create_task ( 'Segment Advisor'
                             , task_id
                             , tname
                             , tname_desc
                             , null
                             );

    dbms_advisor.create_object ( tname
                               , 'TABLESPACE'
                               , 'USERS'
                               , ' '
                               , ' '
                               , null
                               , ' '
                               , object_id
                               );

    dbms_advisor.set_task_parameter ( tname
                               , 'RECOMMEND_ALL'
                               , 'TRUE'
                               );
end;
/

declare
    tname varchar2(128) := 'seg_task_1';
begin
    dbms_advisor.execute_task (tname);
end;
/

select description
from   dba_advisor_tasks
where  task_name = 'seg_task_1'
/
 

==== Where are the logfiles in 13c? ====
  * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=2112329.1|This note shows the directory structure and log file locations of an Enterprise Manager installation]]
==== agent is blocked and needs resynchronization ====
Performing
<code>
emctl status agent
</code>
shows
<code>
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.4.0.0.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /cln/exp/ora_bin1/app/oracle/agent/agent_inst
Agent Log Directory    : /cln/exp/ora_bin1/app/oracle/agent/agent_inst/sysman/log
Agent Binaries         : /cln/exp/ora_bin1/app/oracle/agent/agent_13.4.0.0.0
Core JAR Location      : /cln/exp/ora_bin1/app/oracle/agent/agent_13.4.0.0.0/jlib
Agent Process ID       : 36176236
Parent Process ID      : 46793064
Agent URL              : https://hn1111.cln.be:3872/emd/main/
Local Agent URL in NAT : https://hn1111.cln.be:3872/emd/main/
Repository URL         : https://hn1627.cln.be:4903/empbs/upload
Started at             : 2021-02-08 16:23:08
Started by user        : oracle
Operating System       : AIX version 7.1 (ppc64)
Number of Targets      : 13
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : 2021-02-08 16:23:22
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 86
Size of XML files pending upload(MB)         : 0.28
Available disk space on upload filesystem    : 22.24%
Collection Status                            : Collections enabled
Heartbeat Status                             : Agent is blocked
Blocked Reason         : Plug-in mismatch found between agent and repository. Please perform an agent resync to unblock the agent. Mismatched plug-ins: [ The plug-in is oracle.sysman.si
Last attempted heartbeat to OMS              : 2021-02-08 18:29:26
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2021-02-08 18:32:26
</code>

From the repository
<code>
select * from sysman.mgmt_blocked_agents;
</code>

From Enterprise Manager Cloud Control
  * Navigate to Setup > Manage Cloud Control > Agents 
  * Click problematic agent
  * Click on drop down menu "Agent" and Choose "Resynchronization" in the menu list
Click OK and the resync job will run and unblock the agent once complete. It will take a few minutes to show up as cleared in Enterprise > Summary.

The same thing can be accomplished on the command-line from the oms
<code>
emcli login -username=sysman -password=<password>
emcli sync
emcli resyncAgent -agent="hn1111.cln.be:3872"
</code>
If resync still gives issues, see [[https://support.oracle.com/epmos/faces/DocumentDisplay?parent=DOCUMENT&sourceId=1919191.1&id=1951336.1|MOS Note 1951336.1]]

=== Upgrading the agent on each database host ===
Check the ports needed with this
<code>
emctl status agent
</code>
Download the AgentPull.sh script
<code>
curl "https://hn1627.cln.be:4903/em/install/getAgentImage" --insecure -o /tmp/AgentPull.sh
</code>

Check what agents are available for download from the OMS
<code>
emcli login -username=sysman
emcli sync
emcli get_supported_platforms
or
/tmp/AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=password PLATFORM=AIX -showPlatforms
or
curl --insecure 'https://hn1627.cln.be:4903/em/install/getAgentImage?script=show'
</code>
Download the agent so it can be distributed to the db servers manually (in case curl is not available on the satellites)
<code>
/tmp/AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=password PLATFORM=AIX -download_only
</code>
or manually from the oms
<code>
mkdir /tmp/agentinstaller
emcli get_agentimage -destination=/var/tmp/agentinstaller -platform="IBM AIX on POWER Systems (64-bit)" -version="13.4.0.0.0"
</code>
==== Download the software library catalog from Metalink/MOS if using offline mode and the zip file does not save correctly ====
Hint though: From "Setup : Provisioning and Patching : Offline Patching", right click and save on the zip link instead of left clicking it!
<code>
wget --user=<MOS_LOGIN_USER> --ask-password=on -nv https://updates.oracle.com/download/em_catalog.zip
</code>
Then from "Setup : Extensibility : Self Update", choose "Check Updates" button and follow instructions.\\
When the file is downloaded and transferred onto the oms server...
<code>
wget --http-user=<MOS_user> --http-password=<MOS_password> -O /tmp/p9348486_112000_Generic.zip https://updates.oracle.com/Orion/Download/download_patch/p9348486_112000_Generic.zip
emcli login -username=sysman
emcli sync
emcli import_update_catalog -file=/tmp/p9348486_112000_Generic.zip -omslocal
emcli logout
</code>
This will now show the latest software available on the Self Update page. Now we need to download the 13.4 agent and upload it back into the software library so that it can be distributed.\\
From "Setup : Extensibility : Self Update : Agent Software", choose the required agent and click the Download button. A box will open with offline instructions.
<code>
wget --http-user=<MOS_user> --http-password=<MOS_password> -O /tmp/p30812887_112000_Generic.zip https://updates.oracle.com/Orion/Services/download/p30812887_112000_Generic.zip?aru=23357833&patch_file=p30812887_112000_Generic.zip
emcli login -username=sysman
emcli sync
emcli import_update -omslocal -file=/tmp/p30812887_112000_Generic.zip
emcli logout
</code>
A rough script to upgrade the agent
==== Create a response file ====
<code>
printf "%s" "Enter sysman password: "
read LOGIN_PASSWORD
printf "%s" "Enter agent registration password: "
read AGENT_REGISTRATION_PASSWORD

cat<<EOCAT >/tmp/em_agent.rsp
LOGIN_USER=sysman
LOGIN_PASSWORD=${LOGIN_PASSWORD}
PLATFORM="AIX"
VERSION=13.4.0.0.0
AGENT_REGISTRATION_PASSWORD=${AGENT_REGISTRATION_PASSWORD}
AGENT_BASE_DIR=${ORACLE_BASE}/agent
AGENT_INSTANCE_HOME=${ORACLE_BASE}/agent/agent_inst
AGENT_PORT=3872
EM_UPLOAD_PORT=4903
OMS_HOST=hn1627.cln.be
ORACLE_HOSTNAME=$(hostname)
EOCAT
</code>

==== Source to get variables into current session ====
<code>
. /tmp/em_agent.rsp
</code>

<code>
curl -sk https://${OMS_HOST}:7803/em/install/getAgentImage > /tmp/AgentPull.sh
chmod 755 /tmp/AgentPull.sh
</code>

==== install the agent ====
<code>
/tmp/AgentPull.sh -ignorePrereqs RSPFILE_LOC=/tmp/em_agent.rsp
</code>

==== Post Install steps ====
<code>
$AGENT_HOME/root.sh
</code>


==== Promoting PDB failing with ORA-01422: exact fetch returns more than requested number of rows (Doc ID 2098371.1) ====
A Container database was getting discovered but failed to promote. The status stayed yellow (pending).
== Cause ==
A duplicate entry exists in the repository for the parent of the PDB target (ie. the CDB one).
<pre>
SELECT prnt.target_guid, agt.emd_url, prnt.monitoring_mode, prnt.host_name, prnt.promote_status
FROM em_targets prnt,
mgmt_assoc_instances inst,
mgmt_assoc_type_extends ext,
mgmt_targets agt
WHERE prnt.target_guid = inst.source_me_guid
AND inst.dest_me_guid = (select entity_guid from EM_MANAGEABLE_ENTITIES where entity_name='<PDB target name which you try to promote>' and entity_type='oracle_pdb')
AND inst.assoc_type = ext.assoc_type
AND ext.extended_assoc_type = 'app_composite_contains'
AND agt.emd_url = prnt.emd_url
AND agt.target_type = 'oracle_emd' ;
</pre>
== Solution ==
- Open Monitoring configuration page for the CDB target.

- Remove the "Pluggable Databases" listed there.

- Run an "Sync Pluggable databases"

- Save the changes.


===== Some useful emcli commands =====
  * Some useful emcli commands and a nice document (if only I understood German) [[https://www.doag.org/formes/pubfiles/8937684/CC132_DOAG_2017_-Verwaltung_mit-_EMCLI.pdf]]
These tips and more at [[https://www.doag.org/formes/pubfiles/8937684/CC132_DOAG_2017_-Verwaltung_mit-_EMCLI.pdf|From doag.org - includes different ways of installing agents]]
Login to the cli and sync the client
<code>
emcli login -username=sysman [-password=<pwd>]
emcli sync
</code>
=== List all the agents ===
<code>
emcli get_targets -targets="oracle_emd"
</code>
<code>
Status  Status           Target Type           Target Name
 ID
1       Up               oracle_emd            hn492.cln.be:3872
1       Up               oracle_emd            hn1111.cln.be:3872
1       Up               oracle_emd            hn481.cln.be:3872
1       Up               oracle_emd            hn991.cln.be:3872
1       Up               oracle_emd            hn511.cln.be:3872
1       Up               oracle_emd            hn990.cln.be:3872
1       Up               oracle_emd            hn1627.cln.be:3872
1       Up               oracle_emd            hn482.cln.be:3872
4       Agent Unreachab  oracle_emd            hn5306.cln.be:3872
        le
1       Up               oracle_emd            hn5405.cln.be:3872
1       Up               oracle_emd            hn491.cln.be:3872
1       Up               oracle_emd            hn5410.cln.be:3872
1       Up               oracle_emd            hn5308.cln.be:3872
1       Up               oracle_emd            hn5407.cln.be:3872
1       Up               oracle_emd            hn512.cln.be:3872

</code>
=== List all databases ===
<code>
emcli get_targets -targets="oracle_database"
</code>
<code>
Status  Status           Target Type           Target Name
 ID
1       Up               oracle_database       bkstt
1       Up               oracle_database       credot
1       Up               oracle_database       rpae
1       Up               oracle_database       rpad
1       Up               oracle_database       appli
1       Up               oracle_database       credoa
1       Up               oracle_database       credoe
1       Up               oracle_database       reportt
1       Up               oracle_database       qfp
0       Down             oracle_database       dwhm
1       Up               oracle_database       itsm
1       Up               oracle_database       xbrla
1       Up               oracle_database       dwht
1       Up               oracle_database       qfa
1       Up               oracle_database       rpaa
1       Up               oracle_database       org
1       Up               oracle_database       dwha
1       Up               oracle_database       dwhd
4       Agent Unreachab  oracle_database       crelsast
        le
1       Up               oracle_database       xbrl
1       Up               oracle_database       crelsasp
1       Up               oracle_database       atm
1       Up               oracle_database       ite
1       Up               oracle_database       amlt
</code>
=== Add a database target to Cloud Control via emcli ===
  * [[https://dbaclass.com/article/emcli-command-oem-12c/|Add target database to OEM cloud]]
<code>
emcli add_target -name="ORACLE_SID" -type="oracle_database" -host="$oemhost" -credentials="UserName:dbsnmp;password:dbsnmp_password;Role:Normal" -properties="SID:ORACLE_SID;Port:1521;OracleHome:;MachineName:\"
</code>
e.g.
<code>
emcli add_target -name="adsdd1" -type="oracle_database" -host="hn1627.cln.be" -credentials="UserName:dbsnmp;password:dbsnmp_password;Role:Normal" -properties="SID:adsdd1;Port:3528;OracleHome:/cln/dev/ora_bin/app/oracle/product/19.13/dbhome_1;MachineName:hn1627.cln.be"
</code>
=== Get list of plugins installed on OMS server ===
<code>
./emcli list_plugins_on_server

OMS name is dbadev64.dba.com.sa:4889_Management_Service
Plug-in Name Plugin-id Version [revision]

Oracle Audit Vault oracle.em.soav 12.1.0.4.0
Oracle GoldenGate oracle.fmw.gg 13.1.1.0.0
Oracle Cloud Framework oracle.sysman.cfw 13.2.1.0.0
Oracle Database oracle.sysman.db 13.2.1.0.0
Oracle Fusion Middleware oracle.sysman.emas 13.2.1.0.0
</code>
=== Get list of plugins installed on a Management Agent ===
<code>
./emcli list_plugins_on_agent -agent_names="172.25.224.2:3872"
The Agent URL is https://172.20.204.2:3872/emd/main/ -
Plug-in Name Plugin-id Version [revision]

Oracle Home oracle.sysman.oh 13.2.0.0.0
Oracle Database oracle.sysman.db 13.2.1.0.0
</code>
=== Create blackout ===
<code>
emcli create_blackout -name="DBACLASS_MULTDB_BLACKOUT" -reason="psu patchcing" -add_targets="DBATEST1:oracle_database;DBATEST2:oracle_database" -schedule="duration:30"
</code>
=== Create blackout for a host(target agent) and all this services/database runnin on that server ===
<code>
./emcli create_blackout -name="DBACLASS_HOST_BLACKOUT" -reason="OS PATCHING OUTAGE" -add_targets="localhost.dbaclass.com:host" -schedule="duration:60" -propagate_targets
</code>
=== Get status of a blackout ===
<code>
./emcli get_blackout_details -name="db_blackout_phase1"
</code>
=== Stop a blackout ===
<code>
./emcli stop_blackout -name="db_blackout_phase1"
./emcli get_blackout_details -name="db_blackout_phase1"
</code>
=== Delete a blackout ===
<code>
./emcli delete_blackout -name="db_blackout_phase1"
./emcli get_blackout_details -name="db_blackout_phase1"
</code>
=== Stop and start an agent from OMS server ===
<code>
./emcli stop_agent -agent_name="dbahost-local:3872" -host_username=oracle
./emcli start_agent -agent_name="dbahost-local:3872" -host_username=oracle
</code>
=== Create a Super Administator account ===
<code>
emcli create_user -name="stuart" -password="password" -privilege="SUPER_USER"
</code>
=== Create an Administrator account with some privileges ===
<code>
emcli create_user -name="someonelse" -password="password" 
    -roles="public;EM_USER" 
    -privilege="OPERATOR_ANY_TARGET" 
    -privilege="CONNECT_ANY_VIEW_TARGET" 
    -privilege="CREATE_JOB"
</code>
=== Show all the Administrator accounts ===
<code>
emcli list -resource="Administrators" -columns="USER_NAME,USER_DESCRIPTION"
</code>
=== Show all active sessions ===
<code>
emcli list_active_sessions -details -table
</code>
=== Change a database user password ===
<code>
emcli update_db_password -target_name=prod1 -user_name=stuart -old_password=<oldpwd> -new_password=<newpwd> -retype_new_password=<newpwd> -change_at_target=yes
</code>
=== Unlock a database user account ===
<code>
emcli update_db_account_status -target_name=prod1 -user_name=DBSNMP -action=UNLOCK -connect_as="DBNamedCreds:NC_SYS"
</code>
or
<code>
emcli update_db_account_status -target_name=dwhm -user_name=DBSNMP -action=UNLOCK -connect_as="DBUserName:sys;DBPassword:sys;DBRole:sysdba"
</code>
=== List groups (including dynamic ones) ===
<code>
emcli get_groups
</code>
<code>
Target Name    Target Type
ACC            group
All Instances  group
DEV_TST_BLD    group
EXP            group
KAP            group
</code>
=== Show group members ===
<code>
emcli get_group_members -name=KAP
</code>
<code>
Target Name  Target Type
EBSO         oracle_database
acc_perf     oracle_database
adso         oracle_database
amlk         oracle_database
bilk         oracle_database
clnk         oracle_database
crelsask     oracle_database
dwh_perf     oracle_database
loank        oracle_database
monprodk     oracle_database
orgperf      oracle_database
portalk      oracle_database
reportk      oracle_database
</code>

=== Create a job by copying an existing one ===
For example, there is now a library job called "Change the Password for the Database Monitoring User" (changes the dbsnmp password on one or many databases!).
  * Run it once so that there is a job in the Activity page.
  * Find the internal name of the job by describing it.
<code>
emcli describe_job -name="CHPASSDBSNMP">/tmp/CHPASSDBSNMP.txt
</code>
<code>
cat /tmp/CHPASSDBSNMP.txt

name=CHPASSDBSNMP
type=dbChangeDBPassword
owner=SYSMAN

target_list=upg:oracle_database

variable.new_password=******
variable.retype_new_password=******

schedule.frequency=IMMEDIATE
</code>
  * This text file can now be edited and used to create a job. Leaving out the variable names for password allows Cloud Control to generate a random password.
<code>
vi /tmp/CHPASSDBSNMPALL.txt

name=CHPASSDBSNMPALL.1
type=dbChangeDBPassword
owner=SYSMAN
target_list="All Instances":group
schedule.frequency=IMMEDIATE
</code>

<code>
emcli create_job -input_file=property_file:/tmp/CHPASSDBSNMPALL.txt
</code>

<code>
emcli get_jobs -name="CHPASSDBSNM%"
</code>
or this one schedules a regular jobs to do the password change
<code>
# Current status of the job is ACTIVE.
name=CHANGE DBSNMP PASSWORD EVERYWHERE.1
type=dbChangeDBPassword
owner=SYSMAN

target_list=All Instances:composite

schedule.frequency=MONTHLY
schedule.startTime=2022-01-11 11:11:11
schedule.days=11
schedule.timezone.region=Target/Local
notification.match=EITHER
properties.max_concurrent_executions=10
</code>



# download emcli to add targets
<code>
wget -P /tmp/emcli --quiet --no-check-certificate https://${OMS_HOST}:7803/em/public_lib_download/emcli/kit/emclikit.jar
[[ $? -ne 0 ]] && echo "ERROR: wget is not working, check the version works with ssl" && exit 1
export JAVA_HOME="${OMS_HOME}/oracle_common/jdk/jre/"
export PATH="${JAVA_HOME}:${PATH}"
java -jar /tmp/emcli/emclikit.jar -install_dir=/tmp/emcli
</code>

# configure emcli
<code>
/tmp/emcli/emcli setup -url=https://${OMS_HOST}:7803/em -username="${LOGIN_USER}" -password="${LOGIN_PASSWORD}" -dir="/tmp/emcli/conf" -trustall -noregister -autologin
</code>


# generate target addition statements
<code>
echo "add_target -name=\"linuxlab01.local_listener\" -type=\"oracle_listener\" -host=\"linuxlab01.local\" -properties=\"LsnrName:LISTENER;ListenerOraDir:/app/oracle/product/12.1.0/network/admin;Port:1521;OracleHome:/app/oracle/product/12.1.0;Machine:linuxlab01.local;\"" >> "/tmp/emcli/emclicommands.txt"
ORAENV_ASK=NO
RUNNING_SIDS="$(ps -efo args|grep '[p]mon'|awk -F'ora_pmon_' '! /\+ASM|^$/ {print $NF}'|sort -fu)"
for i in ${RUNNING_SIDS}; do
    export ORACLE_SID="${i}"
    . oraenv >/dev/null 2>&1
    [[ $? -ne 0 ]] && echo "WARNING: CAnnot set environment for ${i}, skipping" && continue
    echo "add_target -name=\"${i}\" -type=\"oracle_database\" -host=\"${ORACLE_HOSTNAME}\" -credentials=\"UserName:dbsnmp;password:dbsnmp;Role:Normal\" -properties=\"SID:${i};Port:${LISTENER_PORT};OracleHome:${ORACLE_HOME};MachineName:${ORACLE_HOSTNAME};\"" >> "/tmp/emcli/emclicommands.txt"
done
</code>


# load up the targets
<code>
/tmp/emcli/emcli login -username="${LOGIN_USER}" -password="${LOGIN_PASSWORD}"
/tmp/emcli/emcli sync
tmp/emcli/emcli argfile /tmp/emcli/emclicommands.txt
</code>


# remove the emcli
<code>
rm -rf /tmp/emcli
</code>

==== Get the agent to autodiscover and upload the basic targets like the agent itself and the host ====
<code>
emctl config agent addinternaltargets
</code>
==== Run autodiscovery on a target host from OMS server ====
First, create a job properties file
<code>
./emcli describe_job_type -job_type=RunAutoDiscovery > inputs.prop
</code
Second, modify this file to specify the relevant host
<code>
# Description: (Optional) The user specified name of the job
name=AUTO_DISC

# Description: (Optional)
The job type for this job
type=RunAutoDiscovery

# Description: (Optional) The user specified description of the job
description=

# Description: The job owner. The job owner is the user who creates the job.
# Default: the logged in user
# The job owner information displayed here is for documentation only and user is not expected to change it.
owner=

# Description: (Optional) The kind of job
# Legal Values: active, library
kind=


# Fill in the target list before submitting.
# For Example:
# target_list=MyTarget:cluster target_list=172.20.294.137:host


# Description: The type of targets to use for this job target
ype=host


# Description: (Optional) Notify the job owner when a selected state occurs
# Allowed Values: SCHEDULED, RUNNING, ACTION_REQUIRED, SUSPENDED, SUCCEEDED, PROBLEMS
notification=

# Description: (Optional) The run time properties for the job
properties=
</code>
Finally, create (run)a job
<code>
./emcli create_job -input_file=property_file:inputs.prop
</code>
Check the job status
<code>
./emcli get_jobs -name=AUTO_DISC
</code>


=== General info on Oracle Enterprise Manager Cloud/Grid Control ===
  * [[http://www.oracle.com/technetwork/oem/grid-control/twp-security-best-practices-133704.pdf|Grid security best practices]]
  * [[https://docs.oracle.com/cd/cloud-control-13.3/EMVWS/EMVWS.pdf|OEM Cloud Control 13c Management Views]]
=== Additional references: ===
  * Note 951076.1: How to Troubleshoot Communication From a Grid Agent to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
  * Note 1089443.1: How to Troubleshoot Communication From the Grid Console (UI) Machine to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
  * Note 1089693.1: How to Troubleshoot Communication From the Oracle Management Service (OMS) to the Grid Control Repository Database in 10g Enterprise Manager Grid Control?

The Enterprise Manager Grid Control includes the following Components:
  * OMS - Management Service - This is responsible for communicating with the user via a GUI and communicating with the OMR
  * OMR - Management Repository - This is a collection of tables owned by the sysman schema that stores all the data collected by the OMA
  * OMA - Management Agent - This is a perl program that runs on the database host (one for all databases on the host) and uploads data to the OMS for storage in the OMR

  * [[https://dbaclass.com/upgrade-oem-cloud-control-from-12c-to-13c/|Upgrade OEM 12C To 13C - dbaclass.com]]
  * [[https://support.oracle.com/epmos/faces/CertifyResults?searchCtx=st%255EANY%257Cpa%255Epi%255E931_Enterprise%2BManager%2BBase%2BPlatform%2B-%2BOMS%257Evi%255E775884%257Epli%255E20_Linux%2Bx86%257Eplvi%255E23258%257Epln%255ELinux%2Bx86%2BRed%2BHat%2BEnterprise%2BLinux%2B6%257E%257C&_afrWindowMode=0&_adf.ctrl-state=egqooixsw_4|Oracle Enterprise Management (OEM) Compatibility Matrix]]
==== A useful set of emcli commands ====
  * [[https://laurent-leturgez.com/2012/06/11/useful-emcli-commands-in-em-cloud-control-12c/|laurent-leturgez.com]]
  * [[https://www.mydbaworld.com/install-em13c-agent-add-targets-using-command-line/|Install EM13c agent and add targets using the Linux command line only]]
  * [[http://facedba.blogspot.com/2018/02/named-credential-management-in-oem-13c.html|Named Credential Management in OEM 13c using EM CLI]]

<code>
emcli list -resource="Administrators"
</code>
<code>
emcli list -resource="Administrators" -help
</code>
<code>
emcli list -resource="Administrators" -columns="USER_NAME,REPOS_ACCOUNT_STATUS"
</code>
<code>
emcli list -resource=Administrators -columns="USER_NAME,USER_TYPE:20,COST_CENTER,CONTACT,DEPARTMENT:30"
</code>
and same thing in a different way
<code>
emcli list -resource=Administrators -columns="USER_NAME,USER_TYPE,COST_CENTER,CONTACT,DEPARTMENT" -colsize="USER_TYPE:20,DEPARTMENT:30"
</code>
Some resources need a bind variable to be passed to get output. A bind variable is created in the resource and then referenced in the command. For example, this command will list all the default preferred credentials for target type oracle_database.
<code>
emcli list -resource="PreferredCredentialsDefault" -bind="TargetType='oracle_database'" -colsize="SetName:15,TargetType:15"
</code>

<code>
emcli get_targets -targets='%database%'
</code>
<code>
emcli list -resource="Targets"  -columns="TARGET_TYPE,TARGET_NAME" -colsize="TARGET_TYPE:25,TARGET_NAME:60" -search="TARGET_TYPE='oracle_database'"
</code>
<code>
emcli list -resource="Targets" -columns="TARGET_TYPE,TARGET_NAME" -colsize="TARGET_TYPE:25,TARGET_NAME:60" -search="TARGET_TYPE='oracle_database'" -search="TYPE_QUALIFIER3='DB'"
</code>
<code>
emcli list -resource=NamedCredentials
</code>
<code>
emcli list -resource=NamedCredentials | awk '{ print $2 $3}'
</code>
<code>
emcli list -resource=NamedCredentials -columns="CredName,CredOwner"
</code>
<code>
emcli list -resource=NamedCredentials -format=name:csv > '/path/filename.csv'
</code>

Create Named Credentials with the scope of one target:

To be able to create a named credential, you need to know the target name (unless you create a global credential), target type and credential type associated with the target type. Let's say I want to create a named credential for my database named "TESTDB". First I need to login to our EM12c server, and list targets named "TESTDB":
<code>
./emcli create_named_credential -auth_target_type=oracle_database \
-cred_scope=Instance -target_type=oracle_database \
-target_name=TESTDB -cred_type=DBCreds -cred_name=NC_INST_SYS_TESTDB \
-attributes="DBUserName:SYS;DBPassword:Hell0W0rld;DBRole:sysdba" -test
</code>

Create GLOBAL credentials: (note -test becomes -test_target_name)
<code>
emcli create_named_credential -auth_target_type=oracle_database \
-cred_scope=GLOBAL -target_type=oracle_database \
-target_name=TESTDB -cred_type=DBCreds -cred_name=NC_GL_TESTDB_SYS \
-attributes="DBUserName:SYS;DBPassword:Hell0W0rld;DBRole:sysdba" -test_target_name=Global
</code>

==== List all credentials stored in the repository ====
Connect to th repository database as sysdba and run this query
<code>
set lines 1000 pages 100
col cred_owner for a10
col cred_name for a60
col cred_type_name for a20
col target_type for a20
col user_name for a10
select cred_owner
,      cred_name
,      cred_type_name
,      target_type
,      user_name
,      decode (cred_scope, 1, 'Global', 2, 'Target') "cred_scope"
,      to_char(trunc(cred_created_date),'DD-Mon-RRRR') "created_on" 
,      to_char(trunc(cred_updated_date),'DD-Mon-RRRR') "updated_on"
from   sysman.em_nc_creds
where  cred_owner != '<SYSTEM>' 
order  BY 1,2,3;
</code>


==== Startup the Management server, central agent and the repository in the correct order ====
  * [[https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.3.1/emadm/starting-and-stopping-oracle-enterprise-manager-13c-cloud-control.html#GUID-E866CE88-ABF0-4DB9-AB0D-25DD453BACE5|Starting and Stopping Oracle Enterprise Manager 13c Cloud Control]]
The Repository database (and listener!) have to be started before the Middleware components can be started.

=== Change sysman password ===
The sysman password needs to be changed with emctl. Changing it from the database is not good enough.\\
Setup the OMS environment (default is $ORACLE_BASE/middleware). 'stop oms -all' is not done before changing the password as the Admin Server needs to be up to make the change.
<code>
cd $ORACLE_HOME/bin
emctl stop oms
emctl stop oms -bip_only
emctl extended oms jvmd stop -all
emctl config oms -change_repos_pwd -old_pwd oldsysmanpassword -new_pwd newsysmanpassword
emctl stop oms -all
emctl start oms
</code>

==== Change dbsnmp password ====
Finally,found a way of doing this without needing to know the old password!
  * [[https://blogs.oracle.com/observability/post/non-dmsnmp-monitoring-credentials|Thank you Desiree :-)]]
<code>
emcli set_monitoring_credential -target_type=oracle_database -target_name="upg" -set_name="DBCredsMonitoring" -cred_type="DBCreds" -attributes="DBUserName:dbsnmp;DBPassword:password;DBRole:Normal"
</code>
Simple when you know how! But the documentation is quite poor in terms of examples and which verb to use for which situation.

Also in the same article...
Adding a target database using EM CLI

When adding a new database, explicitly or through discovery, you can specify non-DBSNMP monitoring credentials for the database:

=== add_target ===
<code>
emcli add_target
-name="database"
-type="oracle_database"
-host="myhost.example.com"
-credentials="UserName:monuser;password:password1;Role:Normal"
-properties="SID:semcli;Port:15091;OracleHome:/oracle; MachineName:smpamp.example.com"
</code> 

=== discover_db ===
<code>
emcli discover_db -input_file=db_discovery_file:db_discovery_file.txt

</code>
The example file, db_discover_file.txt, may have contents like below:
<code>
host_name=myhost.example.com
db_creds=db1:monuser,password1,NORMAL
target_type=oracle_database

</code>

  * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1592390.1|EM 13c: How to Update DBSNMP Password and Change Monitoring Credentials of an Enterprise Manager Cloud Control Target Database Using emcli Command Line (Doc ID 1592390.1)]]
1) Single Database Instance

How to Use update_db_password verb
<code>
emcli login -username=sysman -password=<sysman_password>
./emcli get_targets -target="oracle_database"
emcli update_db_password -target_name="<target_name>" -target_type="oracle_database" -user_name="dbsnmp" -change_at_target=yes -old_password="<old_password>" -new_password="<new_password>" -retype_new_password="<new_password>"
</code>
=> Replace each value in <> in command line with the appropriate values.

Note that if the database has been independently updated for the DBSNMP password, use of this verb must accommodate this fact.
In such a case, the "-change_at_target" option is no longer needed.

Documentation on update_db_password verb:

For EM 13c: 

Enterprise Manager Command Line Interface\\
Release 13.2\\
Chapter 5 Verb Reference\\
5.4 EM CLI Verbs\\
[[https://docs.oracle.com/cd/E24628_01/em.121/e17786/cli_verb_ref.htm#EMCLI1505|5.4.714 update_db_password]]


=== Login to EM on OMS ===
<code>
emcli login -username=sysman [[-password=<sysmanpassword>]]
</code>
=== logout of EM on OMS ===
<code>
emcli logout
</code>
=== Sync emcli with OMS ===
<code>
emcli sync
</code>
=== List promoted targets ===
<code>
emcli get_targets
</code>
=== Delete a specific target ===
<code>
emcli delete_target -name="devtools" -type="oracle_database"
Target "devtools:oracle_database" deleted successfully
</code>
=== Find all agents ===
<code>
emcli get_targets -format=name:csv | grep "oracle_emd"
</code>
=== Delete an agent and ALL its targets ===
<code>
emcli delete_target -name="hn5407.cln.be:3872" -type="oracle_emd" -delete_monitored_targets
</code>
=== Follow a plugin deployment (on the OMS / on an agent) ===
Oracle Database plugin
<code>
emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db
</code>
Oracle Fusion Middleware plugin
<code>
emcli get_plugin_deployment_status -plugin_id=oracle.sysman.emas
</code>
My Oracle Support plugin
<code>
emcli get_plugin_deployment_status -plugin_id=oracle.sysman.mos
</code>

=== Import an update (for example: a plugin update) into the software library ===
<code>
emcli import_update -file="p14018177_112000_Generic.zip" -omslocal
</code>
=== Deploy a plugin on the OMS ===
<code>
emcli deploy_plugin_on_server -plugin=oracle.sysman.db -sys_password=XXXXX
</code>
=== Deploy a plugin on EM agent(s) ===
<code>
emcli deploy_plugin_on_agent -plugin="oracle.sysman.db"  -agent_names="xxxdb-prod1.mydomain.com:3872;xxxdb-prod2.mydomain.com:3872"
</code>
=== List available agents in the library ===
<code>
emcli get_supported_platforms
</code>
=== Download an agent from the library (used for agentDeploy.sh script method) ===
<code>
emcli get_agentimage -destination=/home/oracle -platform="Microsoft Windows x64 (64-bit)" -version="12.1.0.1.0"
</code>
=== Set monitoring credentials for a specific target (example given for an Oracle database instance) ===
<code>
emcli set_credential -target_type=oracle_database -target_name="prod1" -credential_set=DBCredsMonitoring -user=sysman -column="Role:SYSDBA;UserName:sys;password:XXXXX" -monitoring
</code>

==== Uninstall the agent oracle home that registered with inventory ====
If you are getting "agent home is already registered with the inventory", it means there is an entry for this home in OraInventory.xml already and it will need removing before installing to the same directory.\\
Agent Homes are stored in /etc/oragchomelist (Linux, AIX) and /var/opt/oracle/oragchomelist (Solaris).\\
Removing the entry from oragchomelist makes no difference.\\
It's in the Oracle Inventory file – its location can be found in /etc/oraInst.loc (Linux, AIX) and /var/opt/oracle/oraInst.loc (Solaris).
<code>
ORACLE_HOME="/oracle/cloud/core/12.1.0.1.0"
$ORACLE_HOME/oui/bin/runInstaller -silent -detachHome
</code>
==== Get rid of an agent that won't go away in the Grid screens ====
This has the effect of removing everything related to the host in question!\\
On host where agent is running...
<code>
emctl stop agent
</code>
On Grid server...
<code>
sqlplus / as sysdba
exec mgmt_admin.cleanup_agent('<host>><port>');
exit
</code>
On host where agent is running...
<code>
emctl start agent
</code>

==== Get rid of targets that won't go away in the Grid screens: ====
<code>
exec mgmt_admin.delete_target('target_name','target_type')
</code>

See mgmt_targets table in sysman schema for list of known targets.
==== Re-discover targets ====
<code>
cd $ORACLE_HOME/bin
./agentca -d
</code>
==== Agent Unreachable ====
  *  Is the agent running?
<code>
 From SQL*Plus:
</code>
<code>
select username,program from v$session where LOWER (program) like 'emagent%';
</code>
If no rows selected, the agent is not running.
or
<code>
From Unix:
. oraenv
agent11g (how to see [[which databases are running on the machine]])
emctl status agent

ps -ef|grep [[a]]gent
</code>
There should be a handful. If only a few, kill them and restart the agent.
<code>
 emctl start agent
</code>

//Check the agent log in $ORACLE_HOME/sysman/log//
==== Start a host level blackout from command line (blackout has to already have been created via OEM) ====
This could be included in a shell script before patching, for example...
<code>
emctl start blackout server_maint -nodeLevel
</code>

==== Start a host level blackout for a certain duration ====
This starts a blackout from now until now + 8 hours
<code>
emctl start blackout server_maint -nodeLevel -d 08:00
</code>
==== Stop a blackout from command line ====
When patching is done?...
<code>
emctl stop blackout server_maint
</code>

====  [[Backend WLS or EM application seems to be down]]  ====
====  To list targets known to an agent:  ====
<code>
emctl config agent listtargets
</code>
It looks at the file $AGENT_HOME/sysman/emd/targets.xml

Manually add targets by editing this file and running:
<code>
emctl config agent addtargets $AGENT_HOME/sysman/emd/targets.xml
</code>
Check state and upload directories under $AGENT_HOME/sysman/emd for .err files

==== Disable BI Publisher ====
<code>
$OMS_HOME/bin/emctl config oms -disable_bip
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
BI Publisher Server is Down
BI Publisher has been disabled on this host and will not be started with the 'emctl start oms' or 'emctl start oms -bip_only' commands.
Overall result of operations: SUCCESS
</code>

==== Alter Page Timeout ====
To avoid the message "The page has expired. Click OK to continue".
<code>
cd $OMS_HOME/bin
./emctl set property -name oracle.sysman.eml.maxInactiveTime -value -1 -sysman_pwd sysman_password
./emctl stop oms
./emctl start oms
</code>
Disable the feature altogether with '-1' value (it's null by default), or enter number of minutes.

==== Change Repository listener port ====
If the Oracle listener port is changed, the OMS will need to be reconfigured to use the new port.
<code>
emctl stop oms -all
</code>
Start the administration server
<code>
emctl start oms -admin_only
</code>
Update the OMS with the changed listener port
<code>
emctl config oms –store_repos_details -repos_conndesc <connect descriptor> -repos_user sysman
</code>
Stop and restart te OMS
<code>
emctl stop oms -all
emctl start oms
</code>
==== Create 2048 bit SSL Certificate Authority ====
  * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1611578.1|Regenerating OEM 12c/13c SSL certificates with Higher Keystrength and Signature Algorithm (Doc ID 1611578.1)]]
<code>
emctl secure createca -key_strength 2048
emctl stop oms
emctl start oms
emcli login -username=sysman
emcli sync
emcli get_ca_info -details
</code>
Then secure the agents so thay use the new CA [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1390222.1]]\\
Secure the OMS after securing all the Agents
<code>
emctl secure oms -console [-protocol TLSv1]
</code>
Restart everything
<code>
emctl stop oms -all
emctl start oms
</code>
==== Change sysman password ====
This works until 11.1. For a more complete guide see metalink note [[https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=390374477625884&parent=DOCUMENT&sourceId=1111894.1&id=270516.1&_afrWindowMode=0&_adf.ctrl-state=n4wa0tnro_294|270516.1]] or note [[https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=n4wa0tnro_99&_afrLoop=390817620003957|259379.1]]

//For the DB Control Release 11.2 and higher, you need to set the environment variable ORACLE_UNQNAME to the value of the DB_UNIQUE_NAME database parameter.//

Changing this password is easy as long as it is done correctly. It is a three step process.\\
Step 1. Change the password in the traditional manner.
<code>
SQL> alter user sysman identified by &new_password account unlock;
</code>
Step 2. Change the password in the emoms.properties file.
<code>
vi ${AGENT_HOME}/sysman/config/emoms.properties
</code>
Change the following 2 lines by entering the clear text password where the encrypted password is, and set True to False
<code>
oracle.sysman.eml.mntr.emdRepPwd=<new password here>\\
oracle.sysman.eml.mntr.emdRepPwdEncrypted=False\\
</code>
Step 3. Restart the agent. Picks up the new password (and encrypts it)
<code>
emctl restart agent
</code>

==== Change dbsnmp password (This is 10g. There's a better way now! Search again ====
Changing this password is easy as long as it is done correctly. It is a three step process.\\
=== Change the password in the traditional manner ===
<code>
SQL> alter user dbsnmp identified by &new_password account unlock;
</code>
=== Change the password in the targets.xml file ===
<code>
vi ${AGENT_HOME}/sysman/emd/targets.xml
</code>
Change the following line by entering the clear text password where the encrypted password is, and set True to False
<code>
<Property NAME="password" VALUE="<new password>" ENCRYPTED="FALSE"/>
</code>
=== Restart the agent. Picks up the new password (and encrypts it) ===
<code>
emctl restart agent
</code>

==== Quick Checklist ====
borrowed from [[http://oraxprt.com/2013/08/how-to-troubleshoot-communication-from-the-oracle-management-service-oms-to-a-grid-agent-in-10g-enterprise-manager-grid-control.html|oraxprt.com]]
<code>
    Verify that the Agent on the target machine is up and running using:

    cd <AGENT_HOME>/bin
    emctl status agent


    The command should return output such as:

    emctl status agent
    Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
    Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
    —————————————————————
    Agent Version : 10.2.0.5.0
    OMS Version : 10.2.0.5.0
    Protocol Version : 10.2.0.5.0
    Agent Home : /home/oracle/OracleHomes/agent10g
    Agent binaries : /home/oracle/OracleHomes/agent10g
    Agent Process ID : 24465
    Parent Process ID : 24449
    Agent URL : https://agentmachine.domain:1830/emd/main/
    Repository URL : https://omsmachine.domain:1159/em/upload
    Started at : 2010-04-22 15:35:39
    Started by user : oracle
    ….
    —————————————————————


    which indicates that the Agent has started up fine. Also review the <AGENT_HOME>/sysman/log/emagent.nohup to ensure that the Agent is not re-starting frequently, which can affect the OMS to Agent communication.
    Refer to Note 548928.1: Enterprise Manager Grid Control Agent 10g, Process Control (Start, Stop & Status) Troubleshooting Guide
    Verify that the Agent's URL, as seen in the Grid Console -> Setup -> Agent name page is the same as the value configured for the EMD_URL in the <AGENT_HOME>/sysman/config/emd.properties file.
    Refer Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?

1. OMS / Agent Component level issues

    If the Agent machine is configured with DHCP and/or the IP address of the machine has recently changed, the OMS will not be able to communicate with the Agent.
    Refer Note 605009.1: Problem: OMS Cannot Communicate with Agent if IP Address of the Grid Agent Machine is Changed
    If there is a rogue emagent process on the target machine, then the OMS log/trace files could show communication errors. Refer Note 733879.1: Communication: OMS Log/Trace Files Show 'ERROR eml.OMSHandshake processFailure' for Agent Already Removed from Grid Console
    If the Agent is not capable of accepting incoming connection requests from the OMS, then the communication will fail. Refer Note 550452.1: Communication: OMS to Agent Communication Fails with 'IOException in sending Request :: Broken pipe'
    Verify if there are multiple Agents installed / discovered from this machine. Refer to Note 435728.1: Communication: OMS to Agent Communication Fails with "Connection refused" if Multiple Agent Targets are Discovered

2. Hostname/IP Address Resolution Issues

If the OMS and Agent Components are located in separate machines, then the hostname/IP address resolution should work correctly from the OMS to the Agent machine.
Refer Note 763844.1: How to Verify the Hostname/IP Address Resolution Between the 10g Enterprise Manager Grid Control Components?

If the OMS is unable to resolve the hostname / IP address of the Agent machine, the <OMS_HOME>/sysman/log/emoms.trc will show errors such as below, when trying to access the Agent Homepage in the Grid Console:

2010-04-26 12:01:51,405 [[EMUI_12_01_26_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdUploadData.1732 – IOException in sending Request :: No route to host


To verify the Hostname / IP Address resolution from OMS to Agent machine, follow below steps:

    Collect the following details on the Agent machine:
        Hostname and the corresponding IP Address on which the Agent is configured:

        cd <AGENT_HOME>/bin
        emctl status agent
        Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
        Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
        —————————————————————
        Agent Version : 10.2.0.5.0
        OMS Version : 10.2.0.5.0
        Protocol Version : 10.2.0.5.0
        Agent Home : /home/oracle/OracleHomes/agent10g
        Agent binaries : /home/oracle/OracleHomes/agent10g
        Agent Process ID : 24465
        Parent Process ID : 24449
        Agent URL : https://agentmachine.domain:1830/emd/main/
        Repository URL : https://omsmachine.domain:1159/em/upload
        Started at : 2010-04-22 15:35:39
        Started by user : oracle


        The hostname is the one seen in the 'Agent URL' field.
        Obtain the IP address for this hostname using:

        ping <agentmachine.domain>


        Output of these commands:

        ping <IP adddress of the Agent machine>
        ping <hostname.domain of the Agent machine>
        ping <hostname of the Agent machine>
        nslookup <IP adddress of the Agent machine>
        nslookup <hostname.domain of the Agent machine>
        nslookup <hostname of the Agent machine>


    Collect the following details from the OMS machine:

    ping <IP adddress of the Agent machine>
    ping <hostname.domain of the Agent machine>
    ping <hostname of the Agent machine>
    nslookup <IP adddress of the Agent machine>
    nslookup <hostname.domain of the Agent machine>
    nslookup <hostname of the Agent machine>

Compare the output of the above commands on OMS and Agent machines – the outputs should match. If there is a difference or an error, please enlist the help of your System / Network Administrator to correct the configuration in the hosts file or the DNS.

Note:

1. If all the above commands work fine but the OMS still fails to communicate with the Agent, then stop and restart the OMS once to reset the TCP caching

<OMS_HOME>/opmn/bin
opmnctl stopall
<OMS_HOME>/opmn/bin>
opmnctl startall


2. If the Agent machine has multiple NIC cards / IP addresses, the Agent can be bound to a particular hostname / IP address combination using steps in:
Note 390444.1: How to: Tell the agent to listen to only one specific NIC Network Interface Card?

If the hostname / resolution works fine from the OMS to Agent but the communication still fails, then check for the presence of Firewall or Proxy Server in the setup using the steps below.

3. Firewall Setup / Proxy Server Issues

For details about configuring the Firewall and using the Proxy Server for the EM components, refer
Note 1088393.1: How to Verify the Communication Between the 10g Enterprise Manager Grid Control Components via Firewall/Proxy?

    If the Agent port is blocked, then the <OMS_HOME>/sysman/log/emoms.trc will show:

    2008-12-01 11:21:25,535[[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdTargetsList.1767 – CommException:
    Unable to get list of targets from emd-getEmdTargetsList()
    2008-12-01 11:21:25,541 [[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdTargetsList.1769 - Connection timed out oracle.sysman.emSDK.emd.comm.CommException: Connection timed out
    The following error is displayed when trying to look at the Targets -> Agent Host -> Performance page:

    An error has occurred!

    Unable to obtain data for target solaris.oracle.com. The target may be down. Switching to the last 24 hrs view


    Incorrect Proxy server configuration at the OMS side, can cause problems described in
    Note 395717.1: Communication: OMS to Agent Communication Fails With 'Cannot Establish Proxy Connection' Due to Proxy-Related Settings

To verify the communication between OMS to Agent machine, when Firewall / Proxy server is in use:

    Identify the Agent port and URL using the steps in
    Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?
    Test the connectivity to the Agent URL from the OMS machine, using one of the following methods:
        Open a web-browser on the OMS machine and try to access these URL's:

        http://agentmachine.domain:agentport/em/upload
        OR
        https://agentmachine.domain:agentport/em/upload


        The URL must return an output similar to:

        EMAgent10.1.0.2.0
        Congratulations, EMAgent is working!


        Use telnet

        telnet agentmachine.domain <agent port>


        Sample output:

        telnet agentmachine.domain 3872
        Trying 20.20.20.20…
        Connected to agentmachine.domain.
        Escape character is '^]]'.


        If the access to the port is blocked due to a firewall, then the above command will fail with:

        telnet agentmachine.domain 3872
        Trying 20.20.20.20…
        telnet: connect to address 20.20.20.20: Connection refused


        Use wget

        wget <agent http url>
        OR
        wget –no-check-certificate <agent https url>

If any of the above commands fail, please contact your Network Administrator to determine if there is a Firewall / Proxy Server in use and check the configuration.
</code>

==== Uninstall OEM Cloud Control managements agent ====
  *  [[https://docs.oracle.com/cd/E24628_01/install.121/e24089/deinstall_agent.htm#EMADV351|Use command line to uninstall unwanted agents from database target servers]]
<code>
/u01/app/Oracle/core/12.1.0.5.0/perl/bin/perl /u01/app/Oracle/core/12.1.0.5.0/sysman/install/AgentDeinstall.pl -agentHome /u01/app/Oracle/core/12.1.0.5.0/ -skipRemoval
</code>
If things were installed correctly originally, the deinstall should be as easy as
<code>
$AGENT_HOME/perl/bin/perl $AGENT_HOME/sysman/install/AgentDeinstall.pl  -agentHome $AGENT_HOME
</code>
and will produce something like this
<code>
Agent Oracle Home: /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0


NOTE: The agent base directory: /cln/tst/ora_bin1/app/oracle/product/agent12c will be removed after successful deinstallation of agent home.


 DetachHome Command executed:/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0 -waitForCompletion -invPtrLoc /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 62464 MB    Passed
The inventory pointer is located at /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 62464 MB    Passed
The inventory pointer is located at /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
The Oracle home '/cln/tst/ora_bin1/app/oracle/product/agent12c/sbin' could not be updated as it does not exist.


Deinstall Command executed:/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0}" -waitForCompletion -removeAllFiles -invPtrLoc /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 62464 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-11-28_04-47-46PM. Please wait ...Oracle Universal Installer, Version 11.1.0.11.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.

Starting deinstall


Deinstall in progress (Thursday, November 28, 2019 4:47:54 PM CET)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Thursday, November 28, 2019 4:49:20 PM CET)
End of deinstallations
Please check '/cln/tst/ora_bin1/app/oracle/oraInventory/logs/silentInstall2019-11-28_04-47-46PM.log' for more details.

Do you want us to delete the old oracle home [yes/no] :yes

NOTE: The targets monitored by this Management Agent will not be deleted in the Enterprise Manager Repository by this deinstall script. Make sure to delete the targets manually from the Cloud Control Console for a successful deinstallation.
</code>

== References ==

NOTE:235290.1 – Understanding the Enterprise Manager Management Agent 10g 'emd.properties' File
NOTE:358953.1 – What ports are used in communication between the Grid Control OMS and a Management Agent?
NOTE:471842.1 – Understanding Proxy Settings in Enterprise Manager Grid Control

==== Some useful Metalink Master Documents related to Grid Control ====
  * Master Index for Managing Oracle Database and Listener with Grid Control [[ID 1304021.1]]
  * Master Note for 10g Grid Control Agent Process Control (Start, Stop & Status) & Configuration [[ID 1082009.1]]
  * How to Run the RDA against a Grid Control Installation [[ID 1057051.1]]
  * How to Run the RDA against a Grid Control Installation Release 11g [[ID 1190193.1]]
  * Grid Control Target Maintenance: Steps to Diagnose Issues Related to "Agent Unreachable" Status [[ID 271126.1]]
  * Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues [[ID 1086343.1]]
  * Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control [[ID 1202453.1]]
  * Receiving agent unreachable notification emails very often after 10.2.0.4 agent upgrade [[ID 752296.1]]
  * Healthcheck Metric failing for a 10.2.0.4 Target Database with 10.2.0.4 Agent [[ID 602633.1]]