Difference between revisions of "Patching Oracle Database Appliance (ODA)"

From dbawiki
Jump to: navigation, search
Line 1: Line 1:
Oracle Database Appliance (ODA) is a rack mounted "all-in-one" box supplied by Oracle. It contains 2 nodes, 2 networks, 2 power supplies and a bunch of storage accessible to both nodes.<br />
 
It can be virtualised or bare-metal. You can setup single instance databases, RAC-One (active-passive) or RAC (active-active) databases.<br />
 
A whole new set of commands is used to interact with ODA - oakcli.<br />
 
For example to create a database on ODA, it's no longer
 
<pre>
 
dbca
 
</pre>
 
it's
 
<pre>
 
oakcli create database -d <database name>
 
</pre>
 
 
===Patch preparation===
 
===Patch preparation===
 
Show version details of all ODA components
 
Show version details of all ODA components

Revision as of 15:40, 14 December 2016

Patch preparation

Show version details of all ODA components

/opt/oracle/oak/bin/oakcli show version -detail

Patching ODA means patching the O/S, the Grid Infrastructure (GI) and the RDBMS.
Useful ODA documents:


Download latest ODA patch bundle

Latest ODA patches are found in Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)

Download and copy zip files

Download the files on the relevant patch page for your hardware, copy them to wherever you store patches on the server and Read the README.

Copy zip files to /tmp

Copy the zip files from your patch storage area to /tmp on both ODA_BASE nodes.

Check space

Make sure there is enough space in /root, /tmp, $ORACLE_HOME, and /opt for the unzippped / installed files.

Verify ODA version

/opt/oracle/oak/bin/oakcli show version

Unpack zip files

Do not use unzip!

cd /tmp
oakcli unpack –package p25064836_121290_Linux-x86-64_1of2.zip 
oakcli unpack –package p25064836_121290_Linux-x86-64_2of2.zip

Delete zips from /tmp

If unpack -package was successful, files can be removed from /tmp.

rm -rf /tmp/p25064836_121290_Linux-x86-64*

Check what components will be patched

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 -verify

Backup sysctl.conf

export DATESTAMP=`date +%Y%m%d`
cp /etc/sysctl.conf /etc/sysctl.conf_${DATESTAMP}

Patch the ODA infrastructure

Remove any non-standard RPMs

oakcli validate -c ol6upgrade -prechecks

If this check reveals any RPMs, they must be removed prior to patching. Note: Perform check on both nodes, there may be differences!

rpm -e --nodeps libtermcap-devel.x86_64
rpm -e --nodeps yum-downloadonly.noarch
rpm -e --nodeps gpg-pubkey
rpm -e --nodeps check_mk-agent.noarch
rpm -e --nodeps readline-devel.x86_64

Review size of shared repository

oakcli show repo

and add space if necessary

oakcli configure repo srepo -incsize 100G

Ready for patching...

Perform any specifics such as

  • Set database in blackout in OEM
  • Any other monitoring software?
  • Ensure archivelogs are not deleted. They will be needed for application on physical standby later.
  • anything else that could be a problem. These aliases won't work any more as we just uninstalled the readline package so comment them for now.
cd
vi .bash_profile
#alias sqlplus='rlwrap -D2 -ic sqlplus'
#alias dgmgrl='rlwrap -D2 -ic dgmgrl'
#alias rman='rlwrap -D2 -ic rman'
#alias asmcmd='rlwrap -D2 -ic asmcmd'
#alias lsnrctl='rlwrap -D2 -ic lsnrctl'
#alias adrci='rlwrap -D2 -ic adrci'
#alias impdp='rlwrap -D2 -ic impdp'
#alias expdp='rlwrap -D2 -ic expdp'

Stop Snap management

/opt/oracle/smu/bin/smu stop

Shutdown any virtual machines

oakcli stop vm odavmdp1 -force
oakcli stop vm odavmdp2 -force
oakcli stop vm odavmdb3 -force
oakcli stop vm odavmoms1 -force

Shutdown Grid agent / Listeners

export ORACLE_SID=AGENT
ORAENV_ASK=NO
. oraenv
emctl stop agent
export TNS_ADMIN=$ORACLE_HOME/network/admin/
lsnrctl stop LISTENER_DATAGUARD

Shutdown and backup ODA_BASE

Note: Do this on both nodes!

oakcli stop oda_base

cd /OVS
tar -cvzf  oakDom1.odavm01-rac.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1

Optonally reconfigure ODA_BASE

oakcli configure oda_base

Restart ODA_BASE

After tarball has been successfully created, ODA_BASE can be restarted.

oakcli start oda_base

Apply the ODA infrastructure patch

Note: Do NOT run this in parallel on both nodes. The idea is that one of the nodes remains up and usable while patching the other!

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --infra --local

Run post infra patch checks

/opt/oracle/oak/bin/oakcli validate -c ol6upgrade –postchecks
sudo -u root -i
su - grid
$GRID_HOME/bin/crsctl check crs
/opt/oracle/oak/bin/oakcli show version -detail

Patch the Grid infrastructure

Stop the shared repositories

oakcli stop repo srepo
oakcli show repo

Apply the GI patch

Carefully read the known issues before applying the GI Patch.

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0 --gi

Check cluster health

${ORACLE_HOME}/grid/bin/crsctl check crs

Change some file permissions

chmod 755 ${ORACLE_HOME}/grid/bin/osdbagrp ${ORACLE_HOME}/grid/lib/acfsreplcrs.pl

Restore sysctl.conf

Check for higher values of variables.

${ORACLE_HOME}/grid/bin/crsctl check crs

Check cluster health

${ORACLE_HOME}/grid/bin/crsctl check crs

Patch the RDBMS

Apply the database patch

Once the infrastructure and GI patches have been completed successfully, the RDBMS patches can be applied.
It can be applied locally on each node or as a rolling upgrade and both nodes will be patched automatically.

/opt/oracle/oak/bin/oakcli update -patch  12.1.2.9.0  --database <-local>

Verify database versions

oakcli show dbhomes -detail
oakcli show databases -detail

Post ODA patch activities

Restart any physical standby databases

Restart RAC cluster

svrctl start database

Verify all ODA components are now up-to-date

/opt/oracle/oak/bin/oakcli show version -detail

Restart any virtual machines

oakcli start vm odavmdp1
oakcli start vm odavmdp2
oakcli start vm odavmdb3
oakcli start vm odavmoms1

Remount ZFS volumes on virtual machines

Log in to each of the VMs in turn

sudo mount /orasoft
sudo mount /orabackups/RMAN
sudo mount /orabackups/datapump_hist

Restart OMS

login to OMS server

cd /u01/OracleHomes/Middleware/oms/bin
./emctl status oms
./emctl start oms

Check ODA Grid agent

Kill any old processes and restart agent

export ORACLE_SID=agent
ORAENV_ASK=NO
. oraenv
ps -ef | grep emagent
kill -9 <pid where agent is 12.1.0.6>
emctl start agent

Reinstall any previously uninstalled RPMs

Check for later versions?

  • Nagios plugin
cd /orasoft/rpm
yum install check-mk-agent-1.2.6p14-e23d6a3d4540a28a.noarch.rpm
  • Oracle Snap Manager
cd /orasoft/oracle_snapmanager/oracle-smu-1.2.0-ga/Linux/noarch/
rpm -i oracle-smu-1.2.0-27.noarch.rpm
/opt/oracle/smu/bin/smu start
  • other
yum install libtermcap-devel.x86_64
yum install yum-downloadonly.noarch
yum install gpg-pubkey
yum install readline-devel.x86_64

Uncomment any specific Oracle commands

Like these aliases setup for readline wrapping

alias sqlplus='rlwrap -D2 -ic sqlplus'
alias dgmgrl='rlwrap -D2 -ic dgmgrl'
alias rman='rlwrap -D2 -ic rman'
alias asmcmd='rlwrap -D2 -ic asmcmd'
alias lsnrctl='rlwrap -D2 -ic lsnrctl'
alias adrci='rlwrap -D2 -ic adrci'
alias impdp='rlwrap -D2 -ic impdp'
alias expdp='rlwrap -D2 -ic expdp'

Uncomment any jobs that clean up archivelogs

Take databases out of blackout on OEM

...and any other monitoring software

Check public keys

Patching may destroy public keys

ssh admin@oda02-rac -p 8002

If password is requested, keys are corrupt.

Useful queries

As seen on Pythian


prompt ------------------------------------------
prompt Patches applied to both the $OH and the DB
prompt ------------------------------------------
with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
select x.patch_id
,      x.patch_uid
,      x.rollbackable
,      s.status
,      x.description
from   a
,      xmltable('InventoryInstance/patches/*'
          passing a.patch_output
          columns
             patch_id number path 'patchID',
             patch_uid number path 'uniquePatchID',
             description varchar2(80) path 'patchDescription',
             rollbackable varchar2(8) path 'rollbackable'
       ) x
,      dba_registry_sqlpatch s
where  x.patch_id = s.patch_id
and    x.patch_uid = s.patch_uid
--       and s.bundle_series = 'PSU'
/



prompt --------------------------------------------------------
prompt Patches installed into the $OH but not applied to the DB
prompt --------------------------------------------------------
with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
select x.patch_id
,      x.patch_uid
,      x.description
from   a
,      xmltable('InventoryInstance/patches/*'
         passing a.patch_output
         columns
           patch_id number path 'patchID',
           patch_uid number path 'uniquePatchID',
           description varchar2(80) path 'patchDescription'
       ) x
minus
select s.patch_id
,      s.patch_uid
,      s.description
from   dba_registry_sqlpatch s
/




prompt --------------------------------------------------------
prompt Patches applied to the DB but not installed into the $OH
prompt --------------------------------------------------------
with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
select s.patch_id
,      s.patch_uid
,      s.description
from   dba_registry_sqlpatch s
minus
select x.patch_id
,      x.patch_uid
,      x.description
from   a
,      xmltable('InventoryInstance/patches/*'
          passing a.patch_output
          columns
             patch_id number path 'patchID',
             patch_uid number path 'uniquePatchID',
             description varchar2(80) path 'patchDescription'
       ) x
/