* [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=888.1|Primary Note for Database Proactive Patch Program (Doc ID 888.1)]] - Patching master document, or * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=2118136.2|Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (Doc ID 2118136.2)]] Useful ODA documents:\\ * [[https://docs.oracle.com/cd/E80799_01/doc.121/e80525/toc.htm|Database Appliance Administration and Reference Guide (12.1.2.9)]]\\ * [[https://docs.oracle.com/cd/E22693_01/doc.12/e55580/oakcli.htm|Database Appliance Administration and Reference Guide (12.1.2.2)]] * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1417713.2|Master ODA document]] * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=888888.1|Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)]] * [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=1937782.1|12.1.0.2 Proactive Database Bundle Patches]] Bundle patches for ODA are installed in 3 stages.
Upto and including 12.1.2.6, this is performed as so:
oakcli update -patch 12.1.2.6.0 --infra # patches the server firm/software oakcli update -patch 12.1.2.6.0 --gi # patches the Grid (ASM/ACFS) firm/software oakcli update -patch 12.1.2.6.0 --database # patches the ORACLE_HOME(s) software From 12.1.2.7 onwards this has changed to:\\ oakcli update -patch 12.1.2.9.0 --server # patches the server and Grid firm/software oakcli update -patch 12.1.2.9.0 --storage # patches the Shared storage including HDD, SSD, Controller, and Expander oakcli update -patch 12.1.2.9.0 --database # patches the ORACLE_HOME(s) software ==== Patch preparation ==== Show version details of all ODA components\\ **Note: all oakcli commands must be executed as root** /opt/oracle/oak/bin/oakcli show version -detail Patching ODA means patching the O/S, the Grid Infrastructure (GI) and the RDBMS (or any combination thereof).\\ === Download latest ODA patch bundle === Latest ODA patches are found in [[https://support.oracle.com/epmos/faces/DocumentDisplay?id=888888.1|Oracle Database Appliance - 12.1.2 and 2.X Supported ODA Versions & Known Issues (Doc ID 888888.1)]] === Download and copy zip files === Download the files on the relevant patch page for your hardware, copy them to wherever you store patches on the server and Read the README. === Copy zip files to /tmp === Copy the zip files from your patch storage area to /tmp on both ODA_BASE nodes. === Check space === Make sure there is enough space in /root, /tmp, $ORACLE_HOME, and /opt for the unzippped / installed files. === Verify ODA version === /opt/oracle/oak/bin/oakcli show version === Unpack zip files === Do not use unzip! cd /tmp oakcli unpack –package p25064836_121290_Linux-x86-64_1of2.zip oakcli unpack –package p25064836_121290_Linux-x86-64_2of2.zip === Delete zips from /tmp === If unpack -package was successful, files can be removed from /tmp. rm -rf /tmp/p25064836_121290_Linux-x86-64* === Check if the patch is likely to work! === The following command will inform you whether the patch will succeed. If the output indicates a possible problem, for example, an unpacked version of the patch is not available on the system, you may wish to defer applying the patch until you have downtime to fix the expected problem.\\ From Node 0: oakcli validate -c ospatch -ver e.g. oakcli validate -c ospatch -ver 12.1.2.9.0 === Remove any non-standard RPMs === If the validation check reveals any RPMs, they must be removed prior to patching.\\ Note: Perform check on both nodes, there may be differences! rpm -e --nodeps libtermcap-devel.x86_64 rpm -e --nodeps yum-downloadonly.noarch rpm -e --nodeps gpg-pubkey rpm -e --nodeps check_mk-agent.noarch rpm -e --nodeps readline-devel.x86_64 === Backup sysctl.conf === This file will probably be modified by any server patches export DATESTAMP=`date +%Y%m%d` cp /etc/sysctl.conf /etc/sysctl.conf_${DATESTAMP} ==== Patch the ODA infrastructure ==== The oakcli patching process has component specific logging enabled for each component being patched.\\ These log files can be found under the directory /opt/oracle/oak/log/$(hostname)/patch/12.1.2.9.0 === Review size of shared repository === oakcli show repo and add space if necessary oakcli configure repo srepo -incsize 100G === Stop any running jobs === Before updating the repository, agent, server, or database, ensure that there will be no jobs running or pending during the update window. === Ready for patching... === Perform any specifics such as * Set database in blackout in OEM * Any other monitoring software? * Ensure archivelogs are not deleted. They will be needed for application on physical standby later. * anything else that could be a problem. For example, here we use rlwrap for command line editing so these aliases will no longer work as we just uninstalled the readline package so comment them for now. cd vi .bash_profile - alias sqlplus='rlwrap -D2 -ic sqlplus' - alias dgmgrl='rlwrap -D2 -ic dgmgrl' - alias rman='rlwrap -D2 -ic rman' - alias asmcmd='rlwrap -D2 -ic asmcmd' - alias lsnrctl='rlwrap -D2 -ic lsnrctl' - alias adrci='rlwrap -D2 -ic adrci' - alias impdp='rlwrap -D2 -ic impdp' - alias expdp='rlwrap -D2 -ic expdp' If these commands are going to be used, they will need unaliasing before use! e.g. unalias lsnrctl === Stop the Snap Management Utility === /opt/oracle/smu/bin/smu stop === Shutdown any virtual machines === oakcli stop vm odavmdp1 -force oakcli stop vm odavmdp2 -force oakcli stop vm odavmdb3 -force oakcli stop vm odavmoms1 -force === Shutdown Grid agent / Listeners === export ORACLE_SID=AGENT ORAENV_ASK=NO . oraenv emctl stop agent export TNS_ADMIN=$ORACLE_HOME/network/admin/ lsnrctl stop LISTENER_DATAGUARD === Shutdown and backup ODA_BASE === Note: Do this on both nodes! oakcli stop oda_base cd /OVS tar -cvzf oakDom1.odavm01-rac.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1 === Optonally reconfigure ODA_BASE === oakcli configure oda_base === Restart ODA_BASE === After tarball has been successfully created, ODA_BASE can be restarted. oakcli start oda_base === Stop the shared repositories === This was done before applying GI patch but as server patch now includes GI, stop repositories before applying server patch. oakcli stop repo srepo oakcli show repo === List the components that require patching === If any component shows as “Up-to-date” for the supported version, then it indicates that the existing version on the system is current and that component will not be patched. From node 0: /opt/oracle/oak/bin/oakcli update -patch 12.1.2.9.0 --verify === Apply the ODA infrastructure (server) patch === Nodes can be patched one at a time by using the -local switch\\ If both nodes are to be patched automatically (rolling patch), do not specify -local switch but command must then be executed from node 0. From node 0: /opt/oracle/oak/bin/oakcli update -patch --server -local e.g. /opt/oracle/oak/bin/oakcli update -patch 12.1.2.9.0 --server -local Note: If using '-local' switch, do not run this command simultaneously on both nodes! === Run post server patch patch checks === Check cluster is working ok. su - grid export ORACLE_SID=+ASM1 ORAENV_ASK=NO . oraenv ${ORACLE_HOME}/bin/crsctl check crs /opt/oracle/oak/bin/oakcli show version -detail === Check NFS mounts are working === List mount points that //should be// mounted grep nfs /etc/fstab | egrep -e "^#" -v | wc -l List mount points that //are// mounted df -hPt nfs | tail -n +2 | wc -l ==== Patch the Grid infrastructure ==== Note: As of 12.1.2.7.0, this step is included in the '--server' patch\\ === Stop the shared repositories === oakcli stop repo srepo oakcli show repo === Apply the GI patch === **Note: As of 12.1.2.7.0, this step is included in the '--server' patch**\\ Carefully read the known issues before applying the GI Patch. /opt/oracle/oak/bin/oakcli update -patch 12.1.2.6.0 --gi === Check cluster health === ${ORACLE_HOME}/grid/bin/crsctl check crs ==== Patch the Shared Storage ==== **Note: This section is new as of 12.1.2.7.0**\\ === Stop the shared repositories === oakcli stop repo srepo oakcli show repo === Apply the Storage patch === Note: The storage patching has no '-local' option. It must start from node 0, node 0 is patched first followed by the node 1. Both nodes will be rebooted.\\ From node 0: /opt/oracle/oak/bin/oakcli update -patch 12.1.2.9.0 --storage === Check cluster health === ${ORACLE_HOME}/grid/bin/crsctl check crs === Check/change some file permissions === su - grid export ORACLE_SID=+ASM1 ORAENV_ASK=NO . oraenv ls -al ${ORACLE_HOME}/bin/osdbagrp ${ORACLE_HOME}/lib/acfsreplcrs.pl chmod 755 ${ORACLE_HOME}/bin/osdbagrp ${ORACLE_HOME}/lib/acfsreplcrs.pl === Restore sysctl.conf === Check for higher values of variables. ==== Patch the RDBMS ==== === Apply the database patch === Once the infrastructure and GI patches (or server and storage) have been completed successfully, the RDBMS patches can be applied.\\ It can be applied locally on each node or as a rolling upgrade and both nodes will be patched automatically.\\ If '-local' is not specified, the command must be run from node 0. /opt/oracle/oak/bin/oakcli update -patch 12.1.2.9.0 --database <-local> === Verify database versions === oakcli show dbhomes -detail oakcli show databases -detail ==== Post ODA patch activities ==== === Restart any physical standby databases === === Restart RAC cluster === svrctl start database === Verify all ODA components are now up-to-date === /opt/oracle/oak/bin/oakcli show version -detail === Restart any virtual machines === oakcli start vm odavmdp1 oakcli start vm odavmdp2 oakcli start vm odavmdb3 oakcli start vm odavmoms1 === Remount ZFS volumes on virtual machines === Log in to each of the VMs in turn sudo mount /orasoft sudo mount /orabackups/RMAN sudo mount /orabackups/datapump_hist === Restart OMS === login to OMS server cd /u01/OracleHomes/Middleware/oms/bin ./emctl status oms ./emctl start oms === Check ODA Grid agent === Kill any old processes and restart agent export ORACLE_SID=agent ORAENV_ASK=NO . oraenv ps -ef | grep emagent kill -9 emctl start agent === Reinstall any previously uninstalled RPMs === Check for later versions? * Nagios plugin cd /orasoft/rpm yum install check-mk-agent-1.2.6p14-e23d6a3d4540a28a.noarch.rpm * Oracle Snap Management Utility cd /orasoft/oracle_snapmanager/oracle-smu-1.2.0-ga/Linux/noarch/ rpm -i oracle-smu-1.2.0-27.noarch.rpm /opt/oracle/smu/bin/smu start * other yum install libtermcap-devel.x86_64 yum install yum-downloadonly.noarch yum install gpg-pubkey yum install readline-devel.x86_64 === Uncomment any specific Oracle commands === Like these aliases setup for readline wrapping alias sqlplus='rlwrap -D2 -ic sqlplus' alias dgmgrl='rlwrap -D2 -ic dgmgrl' alias rman='rlwrap -D2 -ic rman' alias asmcmd='rlwrap -D2 -ic asmcmd' alias lsnrctl='rlwrap -D2 -ic lsnrctl' alias adrci='rlwrap -D2 -ic adrci' alias impdp='rlwrap -D2 -ic impdp' alias expdp='rlwrap -D2 -ic expdp' === Uncomment any jobs that clean up archivelogs === === Take databases out of blackout on OEM === ...and any other monitoring software === Check public keys === Patching may destroy public keys ssh admin@oda02-rac -p 8002 If password is requested, keys are corrupt. ==== When it all goes tits up ==== === infra patch fails with === [[root@odavm02-rac bin]]# ./crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'odavm02-rac' CRS-2673: Attempting to stop 'ora.crsd' on 'odavm02-rac' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'odavm02-rac' CRS-2679: Attempting to clean 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac' CRS-2680: Clean of 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac' failed CRS-2799: Failed to shut down resource 'ora.DATA.DBSTAGING.advm' on 'odavm02-rac' CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'odavm02-rac' has failed CRS-2675: Stop of 'ora.crsd' on 'odavm02-rac' failed CRS-2799: Failed to shut down resource 'ora.crsd' on 'odavm02-rac' CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'odavm02-rac' has failed CRS-4687: Shutdown command has completed with errors. CRS-4000: Command Stop failed, or completed with errors. * Solution Recreate STAGING volume on ODA_BASE. [[grid@odavm01-rac ~]]$ asmcmd ASMCMD> volcreate -G DATA -s 1024G DBSTAGING ASMCMD> volinfo -G DATA DBSTAGING Diskgroup Name: DATA Volume Name: DBSTAGING Volume Device: /dev/asm/dbstaging-358 State: ENABLED Size (MB): 1048576 Resize Unit (MB): 32 Redundancy: MIRROR Stripe Columns: 4 Stripe Width (K): 128 Usage: Mountpath: [[grid@odavm01-rac ~]]$ /sbin/mkfs -t acfs /dev/asm/dbstaging-358 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/dbstaging-358 mkfs.acfs: volume size = 1099511627776 mkfs.acfs: Format complete. [[grid@odavm01-rac ~]]$ /sbin/acfsutil registry -a -f /dev/asm/dbstaging-358 /staging acfsutil registry: mount point /staging successfully added to Oracle Registry [[grid@odavm01-rac ~]]$ /sbin/acfsutil registry -l /staging Device : /dev/asm/dbstaging-358 : Mount Point : /staging : Options : none : Nodes : all : Disk Group : DATA : Volume : DBSTAGING [[root@odavm01-rac ~]]# /bin/mount -t acfs /dev/asm/dbstaging-358 /staging [[root@odavm01-rac /]]# chown oracle:oinstall /staging [[root@odavm01-rac ~]]# df -h /staging Filesystem Size Used Avail Use% Mounted on /dev/asm/dbstaging-358 1.0T 2.2G 1022G 1% /staging === infra patch fails with === INFO: Running prepatching on node 1 ERROR : Ran '/usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra' and it returned code(1) and output is: INFO: 2015-10-30 21:57:52: checking avaiable free space on /tmp, /u01 and /opt WARNING: 2015-10-30 21:57:52: Free space on / is less than 3G. INFO: 2015-10-30 21:57:52: Please free up space on /. WARNING: 2015-10-30 21:57:52: Free space on / is less than 3G. INFO: 2015-10-30 21:57:52: Please free up space on /. ERROR: 2015-10-30 21:57:52: One or more volumes are running low on free space INFO: 2015-10-30 21:57:52: Please look at the log /opt/oracle/oak/log/odavm02-rac/patch/12.1.2.5.0/prepatch_5974.log for more details error at and errnum=<1> ERROR : Command = /usr/bin/ssh -l root odavm02-rac /opt/oracle/oak/pkgrepos/System/12.1.2.5.0/bin/prepatch -v 12.1.2.5.0 --infra did not complete successfully. Exit code 1 #Step -1# Exiting... ERROR: Failed to apply the patch * Solution Clean the / directory and ensure at least 3G available! === gi patch fails with === [[root@odavm01-rac ~]]# /opt/oracle/oak/bin/oakcli update -patch 12.1.2.5.0 --gi WARNING: One or more shared repos are running. They need to be stopped before patching the GI You have new mail in /var/spool/mail/root [[root@sdtcsynoda01-rac ~]]# oakcli show repo NAME TYPE NODENUM FREE SPACE STATE SIZE odarepo1 local 0 N/A N/A N/A odarepo2 local 1 N/A N/A N/A srepo shared 0 14.88% ONLINE 307200.0M srepo shared 1 14.88% ONLINE 307200.0M * Solution Stop the shared repositories [[root@odavm01-rac ~]]# oakcli stop repo srepo Successfully stopped SharedRepo srepo on node 1 Successfully stopped SharedRepo srepo on node 0 [[root@odavm01-rac ~]]# oakcli show repo NAME TYPE NODENUM FREE SPACE STATE SIZE odarepo1 local 0 N/A N/A N/A odarepo2 local 1 N/A N/A N/A srepo shared 0 N/A OFFLINE N/A srepo shared 1 N/A OFFLINE N/A === If Grid agent does not restart === [[oracle@odavm02-rac SH]]$ emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Status agent Failure:unable to connect to http server at https://odavm02-rac.localwan.net:3872/emd/lifecycle/main/. [[peer not authenticated]] [[oracle@odavm02-rac SH]]$ emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Agent status could not be determined. Check the agent process Consult emctl.log and emagent.nohup in: /u01/app/oracle/agent/12.1.0.4/agent_inst/sysman/log * Problem Grid agent was probably started from wrong ORACLE_HOME * Solution Kill any old agent processes and restart ps -ef | grep emagent kill -9 emctl start agent