cloud_control
Table of Contents
Upgrading the OMS to 13.4
Should the old agent be de-installed before the new one is installed? Seems to work without doing this:
$<AGENT_HOME>/perl/bin/perl <AGENT_HOME>/sysman/install/AgentDeinstall.pl -agentHome <AGENT_HOME>
Create a Segment Advisor task
declare
tname varchar2(128) := 'seg_task_1';
tname_desc varchar2(128) := 'Get shrink advice for segments in tablespace';
task_id number;
object_id number;
object_name varchar2(100);
object_type varchar2(100);
begin
dbms_advisor.create_task ( 'Segment Advisor'
, task_id
, tname
, tname_desc
, null
);
dbms_advisor.create_object ( tname
, 'TABLESPACE'
, 'USERS'
, ' '
, ' '
, null
, ' '
, object_id
);
dbms_advisor.set_task_parameter ( tname
, 'RECOMMEND_ALL'
, 'TRUE'
);
end;
/
declare
tname varchar2(128) := 'seg_task_1';
begin
dbms_advisor.execute_task (tname);
end;
/
select description
from dba_advisor_tasks
where task_name = 'seg_task_1'
/
==== Where are the logfiles in 13c? ====
* [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=2112329.1|This note shows the directory structure and log file locations of an Enterprise Manager installation]]
==== agent is blocked and needs resynchronization ====
Performing
<code>
emctl status agent
shows
Oracle Enterprise Manager Cloud Control 13c Release 4 Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 13.4.0.0.0 OMS Version : (unknown) Protocol Version : 12.1.0.1.0 Agent Home : /cln/exp/ora_bin1/app/oracle/agent/agent_inst Agent Log Directory : /cln/exp/ora_bin1/app/oracle/agent/agent_inst/sysman/log Agent Binaries : /cln/exp/ora_bin1/app/oracle/agent/agent_13.4.0.0.0 Core JAR Location : /cln/exp/ora_bin1/app/oracle/agent/agent_13.4.0.0.0/jlib Agent Process ID : 36176236 Parent Process ID : 46793064 Agent URL : https://hn1111.cln.be:3872/emd/main/ Local Agent URL in NAT : https://hn1111.cln.be:3872/emd/main/ Repository URL : https://hn1627.cln.be:4903/empbs/upload Started at : 2021-02-08 16:23:08 Started by user : oracle Operating System : AIX version 7.1 (ppc64) Number of Targets : 13 Last Reload : (none) Last successful upload : (none) Last attempted upload : 2021-02-08 16:23:22 Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload : 86 Size of XML files pending upload(MB) : 0.28 Available disk space on upload filesystem : 22.24% Collection Status : Collections enabled Heartbeat Status : Agent is blocked Blocked Reason : Plug-in mismatch found between agent and repository. Please perform an agent resync to unblock the agent. Mismatched plug-ins: [ The plug-in is oracle.sysman.si Last attempted heartbeat to OMS : 2021-02-08 18:29:26 Last successful heartbeat to OMS : (none) Next scheduled heartbeat to OMS : 2021-02-08 18:32:26
From the repository
select * from sysman.mgmt_blocked_agents;
From Enterprise Manager Cloud Control
- Navigate to Setup > Manage Cloud Control > Agents
- Click problematic agent
- Click on drop down menu “Agent” and Choose “Resynchronization” in the menu list
Click OK and the resync job will run and unblock the agent once complete. It will take a few minutes to show up as cleared in Enterprise > Summary.
The same thing can be accomplished on the command-line from the oms
emcli login -username=sysman -password=<password> emcli sync emcli resyncAgent -agent="hn1111.cln.be:3872"
If resync still gives issues, see MOS Note 1951336.1
Upgrading the agent on each database host
Check the ports needed with this
emctl status agent
Download the AgentPull.sh script
curl "https://hn1627.cln.be:4903/em/install/getAgentImage" --insecure -o /tmp/AgentPull.sh
Check what agents are available for download from the OMS
emcli login -username=sysman emcli sync emcli get_supported_platforms or /tmp/AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=password PLATFORM=AIX -showPlatforms or curl --insecure 'https://hn1627.cln.be:4903/em/install/getAgentImage?script=show'
Download the agent so it can be distributed to the db servers manually (in case curl is not available on the satellites)
/tmp/AgentPull.sh LOGIN_USER=sysman LOGIN_PASSWORD=password PLATFORM=AIX -download_only
or manually from the oms
mkdir /tmp/agentinstaller emcli get_agentimage -destination=/var/tmp/agentinstaller -platform="IBM AIX on POWER Systems (64-bit)" -version="13.4.0.0.0"
Download the software library catalog from Metalink/MOS if using offline mode and the zip file does not save correctly
Hint though: From “Setup : Provisioning and Patching : Offline Patching”, right click and save on the zip link instead of left clicking it!
wget --user=<MOS_LOGIN_USER> --ask-password=on -nv https://updates.oracle.com/download/em_catalog.zip
Then from “Setup : Extensibility : Self Update”, choose “Check Updates” button and follow instructions.
When the file is downloaded and transferred onto the oms server…
wget --http-user=<MOS_user> --http-password=<MOS_password> -O /tmp/p9348486_112000_Generic.zip https://updates.oracle.com/Orion/Download/download_patch/p9348486_112000_Generic.zip emcli login -username=sysman emcli sync emcli import_update_catalog -file=/tmp/p9348486_112000_Generic.zip -omslocal emcli logout
This will now show the latest software available on the Self Update page. Now we need to download the 13.4 agent and upload it back into the software library so that it can be distributed.
From “Setup : Extensibility : Self Update : Agent Software”, choose the required agent and click the Download button. A box will open with offline instructions.
wget --http-user=<MOS_user> --http-password=<MOS_password> -O /tmp/p30812887_112000_Generic.zip https://updates.oracle.com/Orion/Services/download/p30812887_112000_Generic.zip?aru=23357833&patch_file=p30812887_112000_Generic.zip emcli login -username=sysman emcli sync emcli import_update -omslocal -file=/tmp/p30812887_112000_Generic.zip emcli logout
A rough script to upgrade the agent
Create a response file
printf "%s" "Enter sysman password: "
read LOGIN_PASSWORD
printf "%s" "Enter agent registration password: "
read AGENT_REGISTRATION_PASSWORD
cat<<EOCAT >/tmp/em_agent.rsp
LOGIN_USER=sysman
LOGIN_PASSWORD=${LOGIN_PASSWORD}
PLATFORM="AIX"
VERSION=13.4.0.0.0
AGENT_REGISTRATION_PASSWORD=${AGENT_REGISTRATION_PASSWORD}
AGENT_BASE_DIR=${ORACLE_BASE}/agent
AGENT_INSTANCE_HOME=${ORACLE_BASE}/agent/agent_inst
AGENT_PORT=3872
EM_UPLOAD_PORT=4903
OMS_HOST=hn1627.cln.be
ORACLE_HOSTNAME=$(hostname)
EOCAT
Source to get variables into current session
. /tmp/em_agent.rsp
curl -sk https://${OMS_HOST}:7803/em/install/getAgentImage > /tmp/AgentPull.sh
chmod 755 /tmp/AgentPull.sh
install the agent
/tmp/AgentPull.sh -ignorePrereqs RSPFILE_LOC=/tmp/em_agent.rsp
Post Install steps
$AGENT_HOME/root.sh
Promoting PDB failing with ORA-01422: exact fetch returns more than requested number of rows (Doc ID 2098371.1)
A Container database was getting discovered but failed to promote. The status stayed yellow (pending).
Cause
A duplicate entry exists in the repository for the parent of the PDB target (ie. the CDB one). <pre> SELECT prnt.target_guid, agt.emd_url, prnt.monitoring_mode, prnt.host_name, prnt.promote_status FROM em_targets prnt, mgmt_assoc_instances inst, mgmt_assoc_type_extends ext, mgmt_targets agt WHERE prnt.target_guid = inst.source_me_guid AND inst.dest_me_guid = (select entity_guid from EM_MANAGEABLE_ENTITIES where entity_name='<PDB target name which you try to promote>' and entity_type='oracle_pdb') AND inst.assoc_type = ext.assoc_type AND ext.extended_assoc_type = 'app_composite_contains' AND agt.emd_url = prnt.emd_url AND agt.target_type = 'oracle_emd' ; </pre>
Solution
- Open Monitoring configuration page for the CDB target.
- Remove the “Pluggable Databases” listed there.
- Run an “Sync Pluggable databases”
- Save the changes.
Some useful emcli commands
- Some useful emcli commands and a nice document (if only I understood German) https://www.doag.org/formes/pubfiles/8937684/CC132_DOAG_2017_-Verwaltung_mit-_EMCLI.pdf
These tips and more at From doag.org - includes different ways of installing agents Login to the cli and sync the client
emcli login -username=sysman [-password=<pwd>] emcli sync
List all the agents
emcli get_targets -targets="oracle_emd"
Status Status Target Type Target Name
ID
1 Up oracle_emd hn492.cln.be:3872
1 Up oracle_emd hn1111.cln.be:3872
1 Up oracle_emd hn481.cln.be:3872
1 Up oracle_emd hn991.cln.be:3872
1 Up oracle_emd hn511.cln.be:3872
1 Up oracle_emd hn990.cln.be:3872
1 Up oracle_emd hn1627.cln.be:3872
1 Up oracle_emd hn482.cln.be:3872
4 Agent Unreachab oracle_emd hn5306.cln.be:3872
le
1 Up oracle_emd hn5405.cln.be:3872
1 Up oracle_emd hn491.cln.be:3872
1 Up oracle_emd hn5410.cln.be:3872
1 Up oracle_emd hn5308.cln.be:3872
1 Up oracle_emd hn5407.cln.be:3872
1 Up oracle_emd hn512.cln.be:3872
List all databases
emcli get_targets -targets="oracle_database"
Status Status Target Type Target Name
ID
1 Up oracle_database bkstt
1 Up oracle_database credot
1 Up oracle_database rpae
1 Up oracle_database rpad
1 Up oracle_database appli
1 Up oracle_database credoa
1 Up oracle_database credoe
1 Up oracle_database reportt
1 Up oracle_database qfp
0 Down oracle_database dwhm
1 Up oracle_database itsm
1 Up oracle_database xbrla
1 Up oracle_database dwht
1 Up oracle_database qfa
1 Up oracle_database rpaa
1 Up oracle_database org
1 Up oracle_database dwha
1 Up oracle_database dwhd
4 Agent Unreachab oracle_database crelsast
le
1 Up oracle_database xbrl
1 Up oracle_database crelsasp
1 Up oracle_database atm
1 Up oracle_database ite
1 Up oracle_database amlt
Add a database target to Cloud Control via emcli
emcli add_target -name="ORACLE_SID" -type="oracle_database" -host="$oemhost" -credentials="UserName:dbsnmp;password:dbsnmp_password;Role:Normal" -properties="SID:ORACLE_SID;Port:1521;OracleHome:;MachineName:\"
e.g.
emcli add_target -name="adsdd1" -type="oracle_database" -host="hn1627.cln.be" -credentials="UserName:dbsnmp;password:dbsnmp_password;Role:Normal" -properties="SID:adsdd1;Port:3528;OracleHome:/cln/dev/ora_bin/app/oracle/product/19.13/dbhome_1;MachineName:hn1627.cln.be"
Get list of plugins installed on OMS server
./emcli list_plugins_on_server OMS name is dbadev64.dba.com.sa:4889_Management_Service Plug-in Name Plugin-id Version [revision] Oracle Audit Vault oracle.em.soav 12.1.0.4.0 Oracle GoldenGate oracle.fmw.gg 13.1.1.0.0 Oracle Cloud Framework oracle.sysman.cfw 13.2.1.0.0 Oracle Database oracle.sysman.db 13.2.1.0.0 Oracle Fusion Middleware oracle.sysman.emas 13.2.1.0.0
Get list of plugins installed on a Management Agent
./emcli list_plugins_on_agent -agent_names="172.25.224.2:3872" The Agent URL is https://172.20.204.2:3872/emd/main/ - Plug-in Name Plugin-id Version [revision] Oracle Home oracle.sysman.oh 13.2.0.0.0 Oracle Database oracle.sysman.db 13.2.1.0.0
Create blackout
emcli create_blackout -name="DBACLASS_MULTDB_BLACKOUT" -reason="psu patchcing" -add_targets="DBATEST1:oracle_database;DBATEST2:oracle_database" -schedule="duration:30"
Create blackout for a host(target agent) and all this services/database runnin on that server
./emcli create_blackout -name="DBACLASS_HOST_BLACKOUT" -reason="OS PATCHING OUTAGE" -add_targets="localhost.dbaclass.com:host" -schedule="duration:60" -propagate_targets
Get status of a blackout
./emcli get_blackout_details -name="db_blackout_phase1"
Stop a blackout
./emcli stop_blackout -name="db_blackout_phase1" ./emcli get_blackout_details -name="db_blackout_phase1"
Delete a blackout
./emcli delete_blackout -name="db_blackout_phase1" ./emcli get_blackout_details -name="db_blackout_phase1"
Stop and start an agent from OMS server
./emcli stop_agent -agent_name="dbahost-local:3872" -host_username=oracle ./emcli start_agent -agent_name="dbahost-local:3872" -host_username=oracle
Create a Super Administator account
emcli create_user -name="stuart" -password="password" -privilege="SUPER_USER"
Create an Administrator account with some privileges
emcli create_user -name="someonelse" -password="password"
-roles="public;EM_USER"
-privilege="OPERATOR_ANY_TARGET"
-privilege="CONNECT_ANY_VIEW_TARGET"
-privilege="CREATE_JOB"
Show all the Administrator accounts
emcli list -resource="Administrators" -columns="USER_NAME,USER_DESCRIPTION"
Show all active sessions
emcli list_active_sessions -details -table
Change a database user password
emcli update_db_password -target_name=prod1 -user_name=stuart -old_password=<oldpwd> -new_password=<newpwd> -retype_new_password=<newpwd> -change_at_target=yes
Unlock a database user account
emcli update_db_account_status -target_name=prod1 -user_name=DBSNMP -action=UNLOCK -connect_as="DBNamedCreds:NC_SYS"
or
emcli update_db_account_status -target_name=dwhm -user_name=DBSNMP -action=UNLOCK -connect_as="DBUserName:sys;DBPassword:sys;DBRole:sysdba"
List groups (including dynamic ones)
emcli get_groups
Target Name Target Type ACC group All Instances group DEV_TST_BLD group EXP group KAP group
Show group members
emcli get_group_members -name=KAP
Target Name Target Type EBSO oracle_database acc_perf oracle_database adso oracle_database amlk oracle_database bilk oracle_database clnk oracle_database crelsask oracle_database dwh_perf oracle_database loank oracle_database monprodk oracle_database orgperf oracle_database portalk oracle_database reportk oracle_database
Create a job by copying an existing one
For example, there is now a library job called “Change the Password for the Database Monitoring User” (changes the dbsnmp password on one or many databases!).
- Run it once so that there is a job in the Activity page.
- Find the internal name of the job by describing it.
emcli describe_job -name="CHPASSDBSNMP">/tmp/CHPASSDBSNMP.txt
cat /tmp/CHPASSDBSNMP.txt name=CHPASSDBSNMP type=dbChangeDBPassword owner=SYSMAN target_list=upg:oracle_database variable.new_password=****** variable.retype_new_password=****** schedule.frequency=IMMEDIATE
- This text file can now be edited and used to create a job. Leaving out the variable names for password allows Cloud Control to generate a random password.
vi /tmp/CHPASSDBSNMPALL.txt name=CHPASSDBSNMPALL.1 type=dbChangeDBPassword owner=SYSMAN target_list="All Instances":group schedule.frequency=IMMEDIATE
emcli create_job -input_file=property_file:/tmp/CHPASSDBSNMPALL.txt
emcli get_jobs -name="CHPASSDBSNM%"
or this one schedules a regular jobs to do the password change
# Current status of the job is ACTIVE. name=CHANGE DBSNMP PASSWORD EVERYWHERE.1 type=dbChangeDBPassword owner=SYSMAN target_list=All Instances:composite schedule.frequency=MONTHLY schedule.startTime=2022-01-11 11:11:11 schedule.days=11 schedule.timezone.region=Target/Local notification.match=EITHER properties.max_concurrent_executions=10
# download emcli to add targets
wget -P /tmp/emcli --quiet --no-check-certificate https://${OMS_HOST}:7803/em/public_lib_download/emcli/kit/emclikit.jar
[[ $? -ne 0 ]] && echo "ERROR: wget is not working, check the version works with ssl" && exit 1
export JAVA_HOME="${OMS_HOME}/oracle_common/jdk/jre/"
export PATH="${JAVA_HOME}:${PATH}"
java -jar /tmp/emcli/emclikit.jar -install_dir=/tmp/emcli
# configure emcli
/tmp/emcli/emcli setup -url=https://${OMS_HOST}:7803/em -username="${LOGIN_USER}" -password="${LOGIN_PASSWORD}" -dir="/tmp/emcli/conf" -trustall -noregister -autologin
# generate target addition statements
echo "add_target -name=\"linuxlab01.local_listener\" -type=\"oracle_listener\" -host=\"linuxlab01.local\" -properties=\"LsnrName:LISTENER;ListenerOraDir:/app/oracle/product/12.1.0/network/admin;Port:1521;OracleHome:/app/oracle/product/12.1.0;Machine:linuxlab01.local;\"" >> "/tmp/emcli/emclicommands.txt"
ORAENV_ASK=NO
RUNNING_SIDS="$(ps -efo args|grep '[p]mon'|awk -F'ora_pmon_' '! /\+ASM|^$/ {print $NF}'|sort -fu)"
for i in ${RUNNING_SIDS}; do
export ORACLE_SID="${i}"
. oraenv >/dev/null 2>&1
[[ $? -ne 0 ]] && echo "WARNING: CAnnot set environment for ${i}, skipping" && continue
echo "add_target -name=\"${i}\" -type=\"oracle_database\" -host=\"${ORACLE_HOSTNAME}\" -credentials=\"UserName:dbsnmp;password:dbsnmp;Role:Normal\" -properties=\"SID:${i};Port:${LISTENER_PORT};OracleHome:${ORACLE_HOME};MachineName:${ORACLE_HOSTNAME};\"" >> "/tmp/emcli/emclicommands.txt"
done
# load up the targets
/tmp/emcli/emcli login -username="${LOGIN_USER}" -password="${LOGIN_PASSWORD}"
/tmp/emcli/emcli sync
tmp/emcli/emcli argfile /tmp/emcli/emclicommands.txt
# remove the emcli
rm -rf /tmp/emcli
Get the agent to autodiscover and upload the basic targets like the agent itself and the host
emctl config agent addinternaltargets
Run autodiscovery on a target host from OMS server
First, create a job properties file
./emcli describe_job_type -job_type=RunAutoDiscovery > inputs.prop </code Second, modify this file to specify the relevant host <code> # Description: (Optional) The user specified name of the job name=AUTO_DISC # Description: (Optional) The job type for this job type=RunAutoDiscovery # Description: (Optional) The user specified description of the job description= # Description: The job owner. The job owner is the user who creates the job. # Default: the logged in user # The job owner information displayed here is for documentation only and user is not expected to change it. owner= # Description: (Optional) The kind of job # Legal Values: active, library kind= # Fill in the target list before submitting. # For Example: # target_list=MyTarget:cluster target_list=172.20.294.137:host # Description: The type of targets to use for this job target ype=host # Description: (Optional) Notify the job owner when a selected state occurs # Allowed Values: SCHEDULED, RUNNING, ACTION_REQUIRED, SUSPENDED, SUCCEEDED, PROBLEMS notification= # Description: (Optional) The run time properties for the job properties=
Finally, create (run)a job
./emcli create_job -input_file=property_file:inputs.prop
Check the job status
./emcli get_jobs -name=AUTO_DISC
General info on Oracle Enterprise Manager Cloud/Grid Control
Additional references:
- Note 951076.1: How to Troubleshoot Communication From a Grid Agent to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
- Note 1089443.1: How to Troubleshoot Communication From the Grid Console (UI) Machine to the Oracle Management Service (OMS) in 10g Enterprise Manager Grid Control?
- Note 1089693.1: How to Troubleshoot Communication From the Oracle Management Service (OMS) to the Grid Control Repository Database in 10g Enterprise Manager Grid Control?
The Enterprise Manager Grid Control includes the following Components:
- OMS - Management Service - This is responsible for communicating with the user via a GUI and communicating with the OMR
- OMR - Management Repository - This is a collection of tables owned by the sysman schema that stores all the data collected by the OMA
- OMA - Management Agent - This is a perl program that runs on the database host (one for all databases on the host) and uploads data to the OMS for storage in the OMR
A useful set of emcli commands
emcli list -resource="Administrators"
emcli list -resource="Administrators" -help
emcli list -resource="Administrators" -columns="USER_NAME,REPOS_ACCOUNT_STATUS"
emcli list -resource=Administrators -columns="USER_NAME,USER_TYPE:20,COST_CENTER,CONTACT,DEPARTMENT:30"
and same thing in a different way
emcli list -resource=Administrators -columns="USER_NAME,USER_TYPE,COST_CENTER,CONTACT,DEPARTMENT" -colsize="USER_TYPE:20,DEPARTMENT:30"
Some resources need a bind variable to be passed to get output. A bind variable is created in the resource and then referenced in the command. For example, this command will list all the default preferred credentials for target type oracle_database.
emcli list -resource="PreferredCredentialsDefault" -bind="TargetType='oracle_database'" -colsize="SetName:15,TargetType:15"
emcli get_targets -targets='%database%'
emcli list -resource="Targets" -columns="TARGET_TYPE,TARGET_NAME" -colsize="TARGET_TYPE:25,TARGET_NAME:60" -search="TARGET_TYPE='oracle_database'"
emcli list -resource="Targets" -columns="TARGET_TYPE,TARGET_NAME" -colsize="TARGET_TYPE:25,TARGET_NAME:60" -search="TARGET_TYPE='oracle_database'" -search="TYPE_QUALIFIER3='DB'"
emcli list -resource=NamedCredentials
emcli list -resource=NamedCredentials | awk '{ print $2 $3}'
emcli list -resource=NamedCredentials -columns="CredName,CredOwner"
emcli list -resource=NamedCredentials -format=name:csv > '/path/filename.csv'
Create Named Credentials with the scope of one target:
To be able to create a named credential, you need to know the target name (unless you create a global credential), target type and credential type associated with the target type. Let's say I want to create a named credential for my database named “TESTDB”. First I need to login to our EM12c server, and list targets named “TESTDB”:
./emcli create_named_credential -auth_target_type=oracle_database \ -cred_scope=Instance -target_type=oracle_database \ -target_name=TESTDB -cred_type=DBCreds -cred_name=NC_INST_SYS_TESTDB \ -attributes="DBUserName:SYS;DBPassword:Hell0W0rld;DBRole:sysdba" -test
Create GLOBAL credentials: (note -test becomes -test_target_name)
emcli create_named_credential -auth_target_type=oracle_database \ -cred_scope=GLOBAL -target_type=oracle_database \ -target_name=TESTDB -cred_type=DBCreds -cred_name=NC_GL_TESTDB_SYS \ -attributes="DBUserName:SYS;DBPassword:Hell0W0rld;DBRole:sysdba" -test_target_name=Global
List all credentials stored in the repository
Connect to th repository database as sysdba and run this query
set lines 1000 pages 100 col cred_owner for a10 col cred_name for a60 col cred_type_name for a20 col target_type for a20 col user_name for a10 select cred_owner , cred_name , cred_type_name , target_type , user_name , decode (cred_scope, 1, 'Global', 2, 'Target') "cred_scope" , to_char(trunc(cred_created_date),'DD-Mon-RRRR') "created_on" , to_char(trunc(cred_updated_date),'DD-Mon-RRRR') "updated_on" from sysman.em_nc_creds where cred_owner != '<SYSTEM>' order BY 1,2,3;
Startup the Management server, central agent and the repository in the correct order
The Repository database (and listener!) have to be started before the Middleware components can be started.
Change sysman password
The sysman password needs to be changed with emctl. Changing it from the database is not good enough.
Setup the OMS environment (default is $ORACLE_BASE/middleware). 'stop oms -all' is not done before changing the password as the Admin Server needs to be up to make the change.
cd $ORACLE_HOME/bin emctl stop oms emctl stop oms -bip_only emctl extended oms jvmd stop -all emctl config oms -change_repos_pwd -old_pwd oldsysmanpassword -new_pwd newsysmanpassword emctl stop oms -all emctl start oms
Change dbsnmp password
Finally,found a way of doing this without needing to know the old password!
emcli set_monitoring_credential -target_type=oracle_database -target_name="upg" -set_name="DBCredsMonitoring" -cred_type="DBCreds" -attributes="DBUserName:dbsnmp;DBPassword:password;DBRole:Normal"
Simple when you know how! But the documentation is quite poor in terms of examples and which verb to use for which situation.
Also in the same article… Adding a target database using EM CLI
When adding a new database, explicitly or through discovery, you can specify non-DBSNMP monitoring credentials for the database:
add_target
emcli add_target -name="database" -type="oracle_database" -host="myhost.example.com" -credentials="UserName:monuser;password:password1;Role:Normal" -properties="SID:semcli;Port:15091;OracleHome:/oracle; MachineName:smpamp.example.com"
discover_db
emcli discover_db -input_file=db_discovery_file:db_discovery_file.txt
The example file, db_discover_file.txt, may have contents like below:
host_name=myhost.example.com db_creds=db1:monuser,password1,NORMAL target_type=oracle_database
1) Single Database Instance
How to Use update_db_password verb
emcli login -username=sysman -password=<sysman_password> ./emcli get_targets -target="oracle_database" emcli update_db_password -target_name="<target_name>" -target_type="oracle_database" -user_name="dbsnmp" -change_at_target=yes -old_password="<old_password>" -new_password="<new_password>" -retype_new_password="<new_password>"
⇒ Replace each value in <> in command line with the appropriate values.
Note that if the database has been independently updated for the DBSNMP password, use of this verb must accommodate this fact. In such a case, the “-change_at_target” option is no longer needed.
Documentation on update_db_password verb:
For EM 13c:
Enterprise Manager Command Line Interface
Release 13.2
Chapter 5 Verb Reference
5.4 EM CLI Verbs
5.4.714 update_db_password
Login to EM on OMS
emcli login -username=sysman [[-password=<sysmanpassword>]]
logout of EM on OMS
emcli logout
Sync emcli with OMS
emcli sync
List promoted targets
emcli get_targets
Delete a specific target
emcli delete_target -name="devtools" -type="oracle_database" Target "devtools:oracle_database" deleted successfully
Find all agents
emcli get_targets -format=name:csv | grep "oracle_emd"
Delete an agent and ALL its targets
emcli delete_target -name="hn5407.cln.be:3872" -type="oracle_emd" -delete_monitored_targets
Follow a plugin deployment (on the OMS / on an agent)
Oracle Database plugin
emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db
Oracle Fusion Middleware plugin
emcli get_plugin_deployment_status -plugin_id=oracle.sysman.emas
My Oracle Support plugin
emcli get_plugin_deployment_status -plugin_id=oracle.sysman.mos
Import an update (for example: a plugin update) into the software library
emcli import_update -file="p14018177_112000_Generic.zip" -omslocal
Deploy a plugin on the OMS
emcli deploy_plugin_on_server -plugin=oracle.sysman.db -sys_password=XXXXX
Deploy a plugin on EM agent(s)
emcli deploy_plugin_on_agent -plugin="oracle.sysman.db" -agent_names="xxxdb-prod1.mydomain.com:3872;xxxdb-prod2.mydomain.com:3872"
List available agents in the library
emcli get_supported_platforms
Download an agent from the library (used for agentDeploy.sh script method)
emcli get_agentimage -destination=/home/oracle -platform="Microsoft Windows x64 (64-bit)" -version="12.1.0.1.0"
Set monitoring credentials for a specific target (example given for an Oracle database instance)
emcli set_credential -target_type=oracle_database -target_name="prod1" -credential_set=DBCredsMonitoring -user=sysman -column="Role:SYSDBA;UserName:sys;password:XXXXX" -monitoring
Uninstall the agent oracle home that registered with inventory
If you are getting “agent home is already registered with the inventory”, it means there is an entry for this home in OraInventory.xml already and it will need removing before installing to the same directory.
Agent Homes are stored in /etc/oragchomelist (Linux, AIX) and /var/opt/oracle/oragchomelist (Solaris).
Removing the entry from oragchomelist makes no difference.
It's in the Oracle Inventory file – its location can be found in /etc/oraInst.loc (Linux, AIX) and /var/opt/oracle/oraInst.loc (Solaris).
ORACLE_HOME="/oracle/cloud/core/12.1.0.1.0" $ORACLE_HOME/oui/bin/runInstaller -silent -detachHome
Get rid of an agent that won't go away in the Grid screens
This has the effect of removing everything related to the host in question!
On host where agent is running…
emctl stop agent
On Grid server…
sqlplus / as sysdba
exec mgmt_admin.cleanup_agent('<host>><port>');
exit
On host where agent is running…
emctl start agent
Get rid of targets that won't go away in the Grid screens:
exec mgmt_admin.delete_target('target_name','target_type')
See mgmt_targets table in sysman schema for list of known targets.
Re-discover targets
cd $ORACLE_HOME/bin ./agentca -d
Agent Unreachable
- Is the agent running?
From SQL*Plus:
select username,program from v$session where LOWER (program) like 'emagent%';
If no rows selected, the agent is not running. or
From Unix: . oraenv agent11g (how to see [[which databases are running on the machine]]) emctl status agent ps -ef|grep [[a]]gent
There should be a handful. If only a few, kill them and restart the agent.
emctl start agent
Check the agent log in $ORACLE_HOME/sysman/log
Start a host level blackout from command line (blackout has to already have been created via OEM)
This could be included in a shell script before patching, for example…
emctl start blackout server_maint -nodeLevel
Start a host level blackout for a certain duration
This starts a blackout from now until now + 8 hours
emctl start blackout server_maint -nodeLevel -d 08:00
Stop a blackout from command line
When patching is done?…
emctl stop blackout server_maint
[[Backend WLS or EM application seems to be down]]
To list targets known to an agent:
emctl config agent listtargets
It looks at the file $AGENT_HOME/sysman/emd/targets.xml
Manually add targets by editing this file and running:
emctl config agent addtargets $AGENT_HOME/sysman/emd/targets.xml
Check state and upload directories under $AGENT_HOME/sysman/emd for .err files
Disable BI Publisher
$OMS_HOME/bin/emctl config oms -disable_bip Oracle Enterprise Manager Cloud Control 13c Release 1 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Stopping BI Publisher Server... BI Publisher Server Successfully Stopped BI Publisher Server is Down BI Publisher has been disabled on this host and will not be started with the 'emctl start oms' or 'emctl start oms -bip_only' commands. Overall result of operations: SUCCESS
Alter Page Timeout
To avoid the message “The page has expired. Click OK to continue”.
cd $OMS_HOME/bin ./emctl set property -name oracle.sysman.eml.maxInactiveTime -value -1 -sysman_pwd sysman_password ./emctl stop oms ./emctl start oms
Disable the feature altogether with '-1' value (it's null by default), or enter number of minutes.
Change Repository listener port
If the Oracle listener port is changed, the OMS will need to be reconfigured to use the new port.
emctl stop oms -all
Start the administration server
emctl start oms -admin_only
Update the OMS with the changed listener port
emctl config oms –store_repos_details -repos_conndesc <connect descriptor> -repos_user sysman
Stop and restart te OMS
emctl stop oms -all emctl start oms
Create 2048 bit SSL Certificate Authority
emctl secure createca -key_strength 2048 emctl stop oms emctl start oms emcli login -username=sysman emcli sync emcli get_ca_info -details
Then secure the agents so thay use the new CA https://support.oracle.com/epmos/faces/DocumentDisplay?id=1390222.1
Secure the OMS after securing all the Agents
emctl secure oms -console [-protocol TLSv1]
Restart everything
emctl stop oms -all emctl start oms
Change sysman password
This works until 11.1. For a more complete guide see metalink note 270516.1 or note 259379.1
For the DB Control Release 11.2 and higher, you need to set the environment variable ORACLE_UNQNAME to the value of the DB_UNIQUE_NAME database parameter.
Changing this password is easy as long as it is done correctly. It is a three step process.
Step 1. Change the password in the traditional manner.
SQL> alter user sysman identified by &new_password account unlock;
Step 2. Change the password in the emoms.properties file.
vi ${AGENT_HOME}/sysman/config/emoms.properties
Change the following 2 lines by entering the clear text password where the encrypted password is, and set True to False
oracle.sysman.eml.mntr.emdRepPwd=<new password here>\\ oracle.sysman.eml.mntr.emdRepPwdEncrypted=False\\
Step 3. Restart the agent. Picks up the new password (and encrypts it)
emctl restart agent
Change dbsnmp password (This is 10g. There's a better way now! Search again
Changing this password is easy as long as it is done correctly. It is a three step process.
Change the password in the traditional manner
SQL> alter user dbsnmp identified by &new_password account unlock;
Change the password in the targets.xml file
vi ${AGENT_HOME}/sysman/emd/targets.xml
Change the following line by entering the clear text password where the encrypted password is, and set True to False
<Property NAME="password" VALUE="<new password>" ENCRYPTED="FALSE"/>
Restart the agent. Picks up the new password (and encrypts it)
emctl restart agent
Quick Checklist
borrowed from oraxprt.com
Verify that the Agent on the target machine is up and running using:
cd <AGENT_HOME>/bin
emctl status agent
The command should return output such as:
emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /home/oracle/OracleHomes/agent10g
Agent binaries : /home/oracle/OracleHomes/agent10g
Agent Process ID : 24465
Parent Process ID : 24449
Agent URL : https://agentmachine.domain:1830/emd/main/
Repository URL : https://omsmachine.domain:1159/em/upload
Started at : 2010-04-22 15:35:39
Started by user : oracle
….
—————————————————————
which indicates that the Agent has started up fine. Also review the <AGENT_HOME>/sysman/log/emagent.nohup to ensure that the Agent is not re-starting frequently, which can affect the OMS to Agent communication.
Refer to Note 548928.1: Enterprise Manager Grid Control Agent 10g, Process Control (Start, Stop & Status) Troubleshooting Guide
Verify that the Agent's URL, as seen in the Grid Console -> Setup -> Agent name page is the same as the value configured for the EMD_URL in the <AGENT_HOME>/sysman/config/emd.properties file.
Refer Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?
1. OMS / Agent Component level issues
If the Agent machine is configured with DHCP and/or the IP address of the machine has recently changed, the OMS will not be able to communicate with the Agent.
Refer Note 605009.1: Problem: OMS Cannot Communicate with Agent if IP Address of the Grid Agent Machine is Changed
If there is a rogue emagent process on the target machine, then the OMS log/trace files could show communication errors. Refer Note 733879.1: Communication: OMS Log/Trace Files Show 'ERROR eml.OMSHandshake processFailure' for Agent Already Removed from Grid Console
If the Agent is not capable of accepting incoming connection requests from the OMS, then the communication will fail. Refer Note 550452.1: Communication: OMS to Agent Communication Fails with 'IOException in sending Request :: Broken pipe'
Verify if there are multiple Agents installed / discovered from this machine. Refer to Note 435728.1: Communication: OMS to Agent Communication Fails with "Connection refused" if Multiple Agent Targets are Discovered
2. Hostname/IP Address Resolution Issues
If the OMS and Agent Components are located in separate machines, then the hostname/IP address resolution should work correctly from the OMS to the Agent machine.
Refer Note 763844.1: How to Verify the Hostname/IP Address Resolution Between the 10g Enterprise Manager Grid Control Components?
If the OMS is unable to resolve the hostname / IP address of the Agent machine, the <OMS_HOME>/sysman/log/emoms.trc will show errors such as below, when trying to access the Agent Homepage in the Grid Console:
2010-04-26 12:01:51,405 [[EMUI_12_01_26_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdUploadData.1732 – IOException in sending Request :: No route to host
To verify the Hostname / IP Address resolution from OMS to Agent machine, follow below steps:
Collect the following details on the Agent machine:
Hostname and the corresponding IP Address on which the Agent is configured:
cd <AGENT_HOME>/bin
emctl status agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.5.0
OMS Version : 10.2.0.5.0
Protocol Version : 10.2.0.5.0
Agent Home : /home/oracle/OracleHomes/agent10g
Agent binaries : /home/oracle/OracleHomes/agent10g
Agent Process ID : 24465
Parent Process ID : 24449
Agent URL : https://agentmachine.domain:1830/emd/main/
Repository URL : https://omsmachine.domain:1159/em/upload
Started at : 2010-04-22 15:35:39
Started by user : oracle
The hostname is the one seen in the 'Agent URL' field.
Obtain the IP address for this hostname using:
ping <agentmachine.domain>
Output of these commands:
ping <IP adddress of the Agent machine>
ping <hostname.domain of the Agent machine>
ping <hostname of the Agent machine>
nslookup <IP adddress of the Agent machine>
nslookup <hostname.domain of the Agent machine>
nslookup <hostname of the Agent machine>
Collect the following details from the OMS machine:
ping <IP adddress of the Agent machine>
ping <hostname.domain of the Agent machine>
ping <hostname of the Agent machine>
nslookup <IP adddress of the Agent machine>
nslookup <hostname.domain of the Agent machine>
nslookup <hostname of the Agent machine>
Compare the output of the above commands on OMS and Agent machines – the outputs should match. If there is a difference or an error, please enlist the help of your System / Network Administrator to correct the configuration in the hosts file or the DNS.
Note:
1. If all the above commands work fine but the OMS still fails to communicate with the Agent, then stop and restart the OMS once to reset the TCP caching
<OMS_HOME>/opmn/bin
opmnctl stopall
<OMS_HOME>/opmn/bin>
opmnctl startall
2. If the Agent machine has multiple NIC cards / IP addresses, the Agent can be bound to a particular hostname / IP address combination using steps in:
Note 390444.1: How to: Tell the agent to listen to only one specific NIC Network Interface Card?
If the hostname / resolution works fine from the OMS to Agent but the communication still fails, then check for the presence of Firewall or Proxy Server in the setup using the steps below.
3. Firewall Setup / Proxy Server Issues
For details about configuring the Firewall and using the Proxy Server for the EM components, refer
Note 1088393.1: How to Verify the Communication Between the 10g Enterprise Manager Grid Control Components via Firewall/Proxy?
If the Agent port is blocked, then the <OMS_HOME>/sysman/log/emoms.trc will show:
2008-12-01 11:21:25,535[[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdTargetsList.1767 – CommException:
Unable to get list of targets from emd-getEmdTargetsList()
2008-12-01 11:21:25,541 [[EMUI_11_17_40_/console/admin/rep/emdConfig/emdTargetsMain$target=agentmachine.domain_3A3872$type=oracle*_emd]] ERROR emdConfig.EmdConfigTargetsData getEmdTargetsList.1769 - Connection timed out oracle.sysman.emSDK.emd.comm.CommException: Connection timed out
The following error is displayed when trying to look at the Targets -> Agent Host -> Performance page:
An error has occurred!
Unable to obtain data for target solaris.oracle.com. The target may be down. Switching to the last 24 hrs view
Incorrect Proxy server configuration at the OMS side, can cause problems described in
Note 395717.1: Communication: OMS to Agent Communication Fails With 'Cannot Establish Proxy Connection' Due to Proxy-Related Settings
To verify the communication between OMS to Agent machine, when Firewall / Proxy server is in use:
Identify the Agent port and URL using the steps in
Note 358953.1: What ports are used in communication between the Grid Control OMS and a Management Agent?
Test the connectivity to the Agent URL from the OMS machine, using one of the following methods:
Open a web-browser on the OMS machine and try to access these URL's:
http://agentmachine.domain:agentport/em/upload
OR
https://agentmachine.domain:agentport/em/upload
The URL must return an output similar to:
EMAgent10.1.0.2.0
Congratulations, EMAgent is working!
Use telnet
telnet agentmachine.domain <agent port>
Sample output:
telnet agentmachine.domain 3872
Trying 20.20.20.20…
Connected to agentmachine.domain.
Escape character is '^]]'.
If the access to the port is blocked due to a firewall, then the above command will fail with:
telnet agentmachine.domain 3872
Trying 20.20.20.20…
telnet: connect to address 20.20.20.20: Connection refused
Use wget
wget <agent http url>
OR
wget –no-check-certificate <agent https url>
If any of the above commands fail, please contact your Network Administrator to determine if there is a Firewall / Proxy Server in use and check the configuration.
Uninstall OEM Cloud Control managements agent
/u01/app/Oracle/core/12.1.0.5.0/perl/bin/perl /u01/app/Oracle/core/12.1.0.5.0/sysman/install/AgentDeinstall.pl -agentHome /u01/app/Oracle/core/12.1.0.5.0/ -skipRemoval
If things were installed correctly originally, the deinstall should be as easy as
$AGENT_HOME/perl/bin/perl $AGENT_HOME/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME
and will produce something like this
Agent Oracle Home: /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0
NOTE: The agent base directory: /cln/tst/ora_bin1/app/oracle/product/agent12c will be removed after successful deinstallation of agent home.
DetachHome Command executed:/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0 -waitForCompletion -invPtrLoc /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 62464 MB Passed
The inventory pointer is located at /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 62464 MB Passed
The inventory pointer is located at /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
The Oracle home '/cln/tst/ora_bin1/app/oracle/product/agent12c/sbin' could not be updated as it does not exist.
Deinstall Command executed:/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0}" -waitForCompletion -removeAllFiles -invPtrLoc /cln/tst/ora_bin1/app/oracle/product/agent12c/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 62464 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-11-28_04-47-46PM. Please wait ...Oracle Universal Installer, Version 11.1.0.11.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.
Starting deinstall
Deinstall in progress (Thursday, November 28, 2019 4:47:54 PM CET)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.
Deinstall successful
End of install phases.(Thursday, November 28, 2019 4:49:20 PM CET)
End of deinstallations
Please check '/cln/tst/ora_bin1/app/oracle/oraInventory/logs/silentInstall2019-11-28_04-47-46PM.log' for more details.
Do you want us to delete the old oracle home [yes/no] :yes
NOTE: The targets monitored by this Management Agent will not be deleted in the Enterprise Manager Repository by this deinstall script. Make sure to delete the targets manually from the Cloud Control Console for a successful deinstallation.
References
NOTE:235290.1 – Understanding the Enterprise Manager Management Agent 10g 'emd.properties' File NOTE:358953.1 – What ports are used in communication between the Grid Control OMS and a Management Agent? NOTE:471842.1 – Understanding Proxy Settings in Enterprise Manager Grid Control
Some useful Metalink Master Documents related to Grid Control
- Master Index for Managing Oracle Database and Listener with Grid Control ID 1304021.1
- Master Note for 10g Grid Control Agent Process Control (Start, Stop & Status) & Configuration ID 1082009.1
- How to Run the RDA against a Grid Control Installation ID 1057051.1
- How to Run the RDA against a Grid Control Installation Release 11g ID 1190193.1
- Grid Control Target Maintenance: Steps to Diagnose Issues Related to “Agent Unreachable” Status ID 271126.1
- Master Note for 10g Grid Control Enterprise Manager Communication and Upload issues ID 1086343.1
- Master Note for Target Maintenance Through 10g Enterprise Manager Grid Control ID 1202453.1
- Receiving agent unreachable notification emails very often after 10.2.0.4 agent upgrade ID 752296.1
- Healthcheck Metric failing for a 10.2.0.4 Target Database with 10.2.0.4 Agent ID 602633.1
cloud_control.txt · Last modified: 2023/04/13 19:42 by 127.0.0.1