dbawiki

Show version details of all ODA components
Note: all oakcli commands must be executed as root

/opt/oracle/oak/bin/oakcli show version -detail

Patching ODA means patching the O/S, the Grid Infrastructure (GI) and the RDBMS (or any combination thereof).

Latest ODA patches are found in Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)

Download the files on the relevant patch page for your hardware, copy them to wherever you store patches on the server and Read the README.

Copy the zip files from your patch storage area to /tmp on both ODA_BASE nodes.

Make sure there is enough space in /root, /tmp, $ORACLE_HOME, and /opt for the unzippped / installed files.

/opt/oracle/oak/bin/oakcli show version

Do not use unzip!

cd /tmp
oakcli unpack –package p25064836_121290_Linux-x86-64_1of2.zip
oakcli unpack –package p25064836_121290_Linux-x86-64_2of2.zip

If unpack -package was successful, files can be removed from /tmp.

rm -rf /tmp/p25064836_121290_Linux-x86-64*

The following command will inform you whether the patch will succeed. If the output indicates a possible problem, for example, an unpacked version of the patch is not available on the system, you may wish to defer applying the patch until you have downtime to fix the expected problem.
From Node 0:

oakcli validate -c ospatch -ver <version to be installed>
e.g.
oakcli validate -c ospatch -ver 12.1.2.9.0

If the validation check reveals any RPMs, they must be removed prior to patching.
Note: Perform check on both nodes, there may be differences!

rpm -e --nodeps libtermcap-devel.x86_64
rpm -e --nodeps yum-downloadonly.noarch
rpm -e --nodeps gpg-pubkey
rpm -e --nodeps check_mk-agent.noarch
rpm -e --nodeps readline-devel.x86_64

This file will probably be modified by any server patches

export DATESTAMP=`date +%Y%m%d`
cp /etc/sysctl.conf /etc/sysctl.conf_${DATESTAMP}

The oakcli patching process has component specific logging enabled for each component being patched.
These log files can be found under the directory <tt>/opt/oracle/oak/log/$(hostname)/patch/12.1.2.9.0</tt>

oakcli show repo

and add space if necessary

oakcli configure repo srepo -incsize 100G

Before updating the repository, agent, server, or database, ensure that there will be no jobs running or pending during the update window.

Perform any specifics such as

• Set database in blackout in OEM
• Any other monitoring software?
• Ensure archivelogs are not deleted. They will be needed for application on physical standby later.
• anything else that could be a problem.

For example, here we use rlwrap for command line editing so these aliases will no longer work as we just uninstalled the readline package so comment them for now.

cd
vi .bash_profile
  - alias sqlplus='rlwrap -D2 -ic sqlplus'
  - alias dgmgrl='rlwrap -D2 -ic dgmgrl'
  - alias rman='rlwrap -D2 -ic rman'
  - alias asmcmd='rlwrap -D2 -ic asmcmd'
  - alias lsnrctl='rlwrap -D2 -ic lsnrctl'
  - alias adrci='rlwrap -D2 -ic adrci'
  - alias impdp='rlwrap -D2 -ic impdp'
  - alias expdp='rlwrap -D2 -ic expdp'

If these commands are going to be used, they will need unaliasing before use! e.g.

unalias lsnrctl

/opt/oracle/smu/bin/smu stop

oakcli stop vm odavmdp1 -force
oakcli stop vm odavmdp2 -force
oakcli stop vm odavmdb3 -force
oakcli stop vm odavmoms1 -force

export ORACLE_SID=AGENT
ORAENV_ASK=NO
. oraenv
emctl stop agent
export TNS_ADMIN=$ORACLE_HOME/network/admin/
lsnrctl stop LISTENER_DATAGUARD

Note: Do this on both nodes!

oakcli stop oda_base

cd /OVS
tar -cvzf  oakDom1.odavm01-rac.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1

oakcli configure oda_base

After tarball has been successfully created, ODA_BASE can be restarted.

oakcli start oda_base

This was done before applying GI patch but as server patch now includes GI, stop repositories before applying server patch.

oakcli stop repo srepo
oakcli show repo

If any component shows as “Up-to-date” for the supported version, then it indicates that the existing version on the system is current and that component will not be patched.

From node 0:
/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --verify

Nodes can be patched one at a time by using the -local switch
If both nodes are to be patched automatically (rolling patch), do not specify -local switch but command must then be executed from node 0.

From node 0:
/opt/oracle/oak/bin/oakcli update -patch <patch bundle version> --server -local
e.g.
/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --server -local

Note: If using '-local' switch, do not run this command simultaneously on both nodes!

Check cluster is working ok.

su - grid
export ORACLE_SID=+ASM1
ORAENV_ASK=NO
. oraenv
${ORACLE_HOME}/bin/crsctl check crs
/opt/oracle/oak/bin/oakcli show version -detail

List mount points that should be mounted

grep nfs /etc/fstab | egrep -e "^#" -v | wc -l

List mount points that are mounted

df -hPt nfs | tail -n +2 | wc -l

Note: As of 12.1.2.7.0, this step is included in the '–server' patch

oakcli stop repo srepo
oakcli show repo

Note: As of 12.1.2.7.0, this step is included in the '–server' patch

Carefully read the known issues before applying the GI Patch.

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.6.0 --gi

${ORACLE_HOME}/grid/bin/crsctl check crs

Note: This section is new as of 12.1.2.7.0

oakcli stop repo srepo
oakcli show repo

Note: The storage patching has no '-local' option. It must start from node 0, node 0 is patched first followed by the node 1. Both nodes will be rebooted.
From node 0:

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --storage

${ORACLE_HOME}/grid/bin/crsctl check crs

su - grid
export ORACLE_SID=+ASM1
ORAENV_ASK=NO
. oraenv
ls -al ${ORACLE_HOME}/bin/osdbagrp ${ORACLE_HOME}/lib/acfsreplcrs.pl
chmod 755 ${ORACLE_HOME}/bin/osdbagrp ${ORACLE_HOME}/lib/acfsreplcrs.pl

Check for higher values of variables.

Once the infrastructure and GI patches (or server and storage) have been completed successfully, the RDBMS patches can be applied.
It can be applied locally on each node or as a rolling upgrade and both nodes will be patched automatically.
If '-local' is not specified, the command must be run from node 0.

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0  --database <-local>

oakcli show dbhomes -detail
oakcli show databases -detail

svrctl start database

/opt/oracle/oak/bin/oakcli show version -detail

oakcli start vm odavmdp1
oakcli start vm odavmdp2
oakcli start vm odavmdb3
oakcli start vm odavmoms1

Log in to each of the VMs in turn

sudo mount /orasoft
sudo mount /orabackups/RMAN
sudo mount /orabackups/datapump_hist

login to OMS server

cd /u01/OracleHomes/Middleware/oms/bin
./emctl status oms
./emctl start oms

Kill any old processes and restart agent

export ORACLE_SID=agent
ORAENV_ASK=NO
. oraenv
ps -ef | grep emagent
kill -9 <pid where agent is 12.1.0.6>
emctl start agent

Check for later versions?

• Nagios plugin

cd /orasoft/rpm
yum install check-mk-agent-1.2.6p14-e23d6a3d4540a28a.noarch.rpm

• Oracle Snap Management Utility

cd /orasoft/oracle_snapmanager/oracle-smu-1.2.0-ga/Linux/noarch/
rpm -i oracle-smu-1.2.0-27.noarch.rpm
/opt/oracle/smu/bin/smu start

• other

yum install libtermcap-devel.x86_64
yum install yum-downloadonly.noarch
yum install gpg-pubkey
yum install readline-devel.x86_64

Like these aliases setup for readline wrapping

alias sqlplus='rlwrap -D2 -ic sqlplus'
alias dgmgrl='rlwrap -D2 -ic dgmgrl'
alias rman='rlwrap -D2 -ic rman'
alias asmcmd='rlwrap -D2 -ic asmcmd'
alias lsnrctl='rlwrap -D2 -ic lsnrctl'
alias adrci='rlwrap -D2 -ic adrci'
alias impdp='rlwrap -D2 -ic impdp'
alias expdp='rlwrap -D2 -ic expdp'

…and any other monitoring software

Patching may destroy public keys

ssh admin@oda02-rac -p 8002

If password is requested, keys are corrupt.

[[root@odavm02-rac bin]]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'odavm02-rac'
CRS-2673: Attempting to stop 'ora.crsd' on 'odavm02-rac'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'odavm02-rac'
CRS-2679: Attempting to clean 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac'
CRS-2680: Clean of 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac' failed
CRS-2799: Failed to shut down resource 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'odavm02-rac' has failed
CRS-2675: Stop of 'ora.crsd' on 'odavm02-rac' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'odavm02-rac'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'odavm02-rac' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.

• Solution

Recreate STAGING volume on ODA_BASE.

[[grid@odavm01-rac ~]]$ asmcmd
ASMCMD> volcreate -G DATA -s 1024G DBSTAGING
ASMCMD> volinfo -G DATA  DBSTAGING
Diskgroup Name: DATA

         Volume Name: DBSTAGING
         Volume Device: /dev/asm/dbstaging-358
         State: ENABLED
         Size (MB): 1048576
         Resize Unit (MB): 32
         Redundancy: MIRROR
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage:
         Mountpath:


[[grid@odavm01-rac ~]]$ /sbin/mkfs -t acfs /dev/asm/dbstaging-358
mkfs.acfs: version                   = 11.2.0.4.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/dbstaging-358
mkfs.acfs: volume size               = 1099511627776
mkfs.acfs: Format complete.


[[grid@odavm01-rac ~]]$ /sbin/acfsutil registry -a -f /dev/asm/dbstaging-358 /staging
acfsutil registry: mount point /staging successfully added to Oracle Registry


[[grid@odavm01-rac ~]]$ /sbin/acfsutil registry -l /staging
Device : /dev/asm/dbstaging-358 : Mount Point : /staging : Options : none : Nodes : all : Disk Group : DATA : Volume : DBSTAGING


[[root@odavm01-rac ~]]# /bin/mount -t acfs /dev/asm/dbstaging-358 /staging

[[root@odavm01-rac /]]# chown oracle:oinstall /staging

[[root@odavm01-rac ~]]# df -h /staging
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/dbstaging-358
                      1.0T  2.2G 1022G   1% /staging

INFO: Running  prepatching on node 1
ERROR  : Ran '/usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra' and it returned code(1) and output is:
 INFO: 2015-10-30 21:57:52: checking avaiable free space on /tmp, /u01 and /opt
 WARNING: 2015-10-30 21:57:52: Free space on / is less than 3G.
 INFO: 2015-10-30 21:57:52: Please free up space on /.
 WARNING: 2015-10-30 21:57:52: Free space on / is less than 3G.
 INFO: 2015-10-30 21:57:52: Please free up space on /.
 ERROR: 2015-10-30 21:57:52: One or more volumes are running low on free space
 INFO: 2015-10-30 21:57:52: Please look at the log /opt/oracle/oak/log/odavm02-rac/patch/12.1.2.5.0/prepatch_5974.log for more details

error at<Command = /usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra> and errnum=<1>
ERROR  : Command = /usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra did not complete successfully. Exit code 1 #Step -1#
Exiting...
ERROR: Failed to apply the patch

• Solution

Clean the / directory and ensure at least 3G available!

[[root@odavm01-rac ~]]# /opt/oracle/oak/bin/oakcli update -patch  12.1.2.5.0 --gi
WARNING: One or more shared repos are running. They need to be stopped before patching the GI
You have new mail in /var/spool/mail/root
[[root@sdtcsynoda01-rac ~]]# oakcli show repo

          NAME                          TYPE            NODENUM  FREE SPACE     STATE           SIZE
          odarepo1                      local           0               N/A     N/A             N/A
          odarepo2                      local           1               N/A     N/A             N/A
          srepo                         shared          0            14.88%     ONLINE          307200.0M
          srepo                         shared          1            14.88%     ONLINE          307200.0M

• Solution

Stop the shared repositories

[[root@odavm01-rac ~]]# oakcli stop repo srepo

Successfully stopped SharedRepo srepo on node 1
Successfully stopped SharedRepo srepo on node 0

[[root@odavm01-rac ~]]# oakcli show repo

          NAME                          TYPE            NODENUM  FREE SPACE     STATE           SIZE
          odarepo1                      local           0               N/A     N/A             N/A
          odarepo2                      local           1               N/A     N/A             N/A
          srepo                         shared          0               N/A     OFFLINE         N/A
          srepo                         shared          1               N/A     OFFLINE         N/A

[[oracle@odavm02-rac SH]]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Status agent Failure:unable to connect to http server at https://odavm02-rac.localwan.net:3872/emd/lifecycle/main/. [[peer not authenticated]]


[[oracle@odavm02-rac SH]]$ emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent status could not be determined.  Check the agent process
Consult emctl.log and emagent.nohup in: /u01/app/oracle/agent/12.1.0.4/agent_inst/sysman/log

• Problem

Grid agent was probably started from wrong ORACLE_HOME

• Solution

Kill any old agent processes and restart

ps -ef | grep emagent
kill -9 <emagent processes>
emctl start agent