Written by Nassyam Basha | Introduction This article explains how to perform PSU patch on GI home with different version/release 12.1.0.2 and the RDBMS is in different version 12.1.0.1 and Grid infrastructure management repository is the part of GI home which is introduced in 12c. In this process we also see how to resolve the conflicts and understand which overlay patch to apply in order to fix the specific bug patch which was already installed. Patching Plan and the Environment This is very much important to reader to understand the environment to get idea whether this articles useful to refer or not. This is two nodes RAC cluster (ORA-R2N1, ORA-R2N2) and the database name HADB. The GI home is on 12.1.0.2 and the Database Home is 12.1.0.1. By using opatchauto we can run from root user and hence the patch will be applied on GI + RDBMS homes but here the GI home and RDBMS versions/releases are different, hence we have to deal the GI patch separately and RDBMS separately. On top of that Global Infrastructure Management Repository instance is must start from 12.1.0.2 and it is mandatory for the future upgrades of GI to 12.2 for example. Below is the list of the patches I have applied, however am not including the conflict patches and overlay patches information because it depends purely based on the environment and the patches which are already installed on the homes. Basically this article gives idea to you how to apply PSU on GI and DB homes are different and the GIMR is part of the configuration. OPatch patch of version 12.1.0.1.10 for Oracle software releases 12.1.0.x (NOV 2015) Patch 21951844 - Database Patch Set Update 12.1.0.1.160119 (Jan2016) Patch 22191349 - Oracle Grid Infrastructure Patch Set Update 12.1.0.2.160119 (Jan2016) What is GIMR? GIMR container stores the diagnostic information collected by the cluster health monitor which helps to identify cluster ware related and operating system related problems. This is also a database but runs with the GI home but not from the RDBMS home and there is more flexibility to start the database from any node in case of failure at one node. Of course this option is available from 12.1.0.1 and it is upto you whether you prefer to install or not, if we preferred to install then cluster health monitor enabled to gather the diagnostic data. When we select the GIMR database option then you can see the instance will be created with the name “MGMTDB” and the instance name as “-MGMTDB”, There may be many confusions where the database will be created and so on? Yes that is expected. When we initiate OUI by default it picks the location of OCR/vote disk and still OUI allows flexibility to create database in any other disk groups as well. [root@ORA-R2N1 patches]# ps -ef|grep pmon oragrid 7388 1 0 08:02 ? 00:00:00 asm_pmon_+ASM1 oracle 9315 1 0 08:03 ? 00:00:00 ora_pmon_hadb1 oragrid 12516 1 0 08:03 ? 00:00:00 mdb_pmon_-MGMTDB root 14623 40282 0 08:04 pts/1 00:00:00 grep --color=auto pmon [root@ORA-R2N1 patches]# SQL> select name,db_unique_name,open_mode from v$database; NAME DB_UNIQUE_NAME OPEN_MODE --------- ------------------------------ -------------------- _MGMTDB _mgmtdb READ WRITE Patching the environment We will start first patching the GI home, To use opatchauto to apply patch on oracle homes, then we must have root access, in case root access is not allowed then the patch should apply manually and these steps are available in readme file of the patch. Prior to patching there are few prerequisites which needs to complete them and they are below. If the GI and RDBMS are in same version then one single opatchauto command from GI home will take care everything, but we are into two different patchset hence we have to handle patching individually. Opatch Upgrade Opatch is an utility to apply the patches and this opatch version should meet the requirement as per the patch readme file, the opatch will be available to download from MOS 6880880 and available with various versions and releases, we need to select the appropriate opatch. For example to apply the Jan2016 PSU the opatch version should be 12.1.0.1.7 but the version can be same or higher than that. After downloading we can unzip and then copy under GI_HOME, so that Oracle accesses the latest patch and the version should meets the requirement. [oracle@ORA-R2N1 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH [oracle@ORA-R2N1 ~]$ opatch version OPatch Version: 12.1.0.1.3 OPatch succeeded. [oracle@ORA-R2N1 ~]$ cd /home/oragrid/patches [root@ORA-R2N1 patches]# ls -ltr total 1214032 drwxrwxr-x 4 oragrid oinstall 45 Jul 7 2015 16870214 -rw-r----- 1 oragrid oinstall 481946 Jul 7 2015 p16870214_121020_Linux-x86-64.zip drwxr-x--- 10 oragrid oinstall 4096 Nov 13 15:16 OPatch -rw-r--r-- 1 oragrid oinstall 52774823 Nov 17 15:18 p6880880_121010_Linux-x86-64.zip [oracle@ORA-R2N1 ~]$ unzip p6880880_121010_Linux-x86-64.zip .. . [oracle@ORA-R2N1 ~]$ opatch version OPatch Version: 12.1.0.1.10 OPatch succeeded. Backup of Oracle Home Always it is best practice to take the backup of the Oracle Homes and it can be useful in case any unresolvable issues during the patching in order to avoid the downtime. This is applicable for both GI home and also RDBMS home. The backup can take using the Tar or zip binary. As we have two nodes there is no necessity to take backup from both nodes. One node backup of binaries can be cloned anytime. [root@ORA-R2N1 patches]# crsctl stop cluster -all CRS-2673: Attempting to stop 'ora.crsd' on 'ORA-R2N1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.REDODG4.dg' on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.DATADG1.dg' on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.FRADG2.dg' on 'ORA-R2N1' . . . . . CRS-2673: Attempting to stop 'ora.cssd' on 'ORA-R2N1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ORA-R2N2' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'ORA-R2N2' CRS-2677: Stop of 'ora.cssd' on 'ORA-R2N1' succeeded CRS-2677: Stop of 'ora.cssd' on 'ORA-R2N2' succeeded [root@ORA-R2N1 patches]# [root@ORA-R2N1 root]# cd /u01/app/oracle/product/11.2.0/dbhome_1 [root@ORA-R2N1 dbhome_1]# tar -zcvf /backup/GI_DB_BKP_03_APR_CR1046883/GI12102.tgz . ./.patch_storage/20831110_Jul_11_2015_00_45_40/original_patch/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/backup/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/original_patch/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/assistants/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/assistants/dbca/ ./.patch_storage/21359755_Oct_21_2015_02_52_58/scratch/patched/assistants/dbca/jlib/ ./.patch_storage/21948354_Dec_20_2015_23_39_33/ ./.patch_storage/21948354_Dec_20_2015_23_39_33/backup/ ./.patch_storage/21948354_Dec_20_2015_23_39_33/original_patch/ ./.patch_storage/interim_inventory.txt ./.patch_storage/record_inventory.txt ./.patch_storage/patch_free [root@ORA-R2N2 grid]# $ pwd /backup/GI_DB_BKP_03_APR_CR1046883 oracle@ORA-R2N2:ORA-R2N2:/backup/GI_DB_BKP_03_APR_CR1046883 > $ ls -ltr total 7604336 -rw-r--r-- 1 root root 5016080737 Apr 3 03:21 db12101.tgz -rw-r--r-- 1 root root 2770746668 Apr 3 03:47 GI12102.tgz oracle@ORA-R2N2:/backup/GI_DB_BKP_03_APR_CR1046883 > $ Now we can start the cluster service again for the patching. [root@ORA-R2N1 patches]# crsctl start cluster -all CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ORA-R2N1' CRS-2672: Attempting to start 'ora.evmd' on 'ORA-R2N2' CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ORA-R2N2' CRS-2672: Attempting to start 'ora.evmd' on 'ORA-R2N1' CRS-2676: Start of 'ora.cssdmonitor' on 'ORA-R2N1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'ORA-R2N1' . . . . . . CRS-2672: Attempting to start 'ora.storage' on 'ORA-R2N2' CRS-2676: Start of 'ora.storage' on 'ORA-R2N2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'ORA-R2N2' CRS-2676: Start of 'ora.crsd' on 'ORA-R2N2' succeeded [root@ORA-R2N1 patches]# Patch conflict detection During the plan phase we can run the opatchauto using the analyze option, so that opatchauto will not update system, but it just checks conflicts with the patch and the oracle home. If there are any conflicts then these should be resolved by requesting merge patches. If the conflict patch contains in the superset patch which we are applying then it can be ignored. [root@ORA-R2N1 patches]# opatchauto apply /home/oragrid/patches/22191349 -analyze OPatch Automation Tool Copyright (c)2014, Oracle Corporation. All rights reserved. OPatchauto Version : 12.1.0.1.10 OUI Version : 12.1.0.2.0 Running from : /u01/app/oragrid/product/12.1.0/grid opatchauto log file: /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatchauto/22191349/opatch_gi_2016-04-03_06-32-01_analyze.log NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system. OCM RSP file has been ignored in analyze mode. Parameter Validation: Successful Configuration Validation: Successful Patch Location: /home/oragrid/patches/22191349 Grid Infrastructure Patch(es): 21436941 21948341 21948344 21948354 DB Patch(es): 21948344 21948354 Patch Validation: Successful Grid Infrastructure home: /u01/app/oragrid/product/12.1.0/grid Analyzing patch(es) on "/u01/app/oragrid/product/12.1.0/grid" ... Patch "/home/oragrid/patches/22191349/21436941" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply. Patch "/home/oragrid/patches/22191349/21948341" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply. Patch "/home/oragrid/patches/22191349/21948344" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply. Patch "/home/oragrid/patches/22191349/21948354" analyzed on "/u01/app/oragrid/product/12.1.0/grid" with warning for apply. Apply Summary: opatchauto ran into some warnings during analyze (Please see log file for details): GI Home: /u01/app/oragrid/product/12.1.0/grid: 21436941,21948341,21948344,21948354 opatchauto completed with warnings. [root@ORA-R2N1 patches]# As we see above status of the opatchauto analyze is completed with warnings where we expected to be succeeded. We can refer the log files for more information whether the conflicts need to be resolved or the patch which will be rollback is part of the patch we are applying. [Apr 3, 2016 1:58:39 AM] Whole composite patch Conflicts/Supersets are: Composite Patch : 21948354 Bug Superset of 16870214 Detail Conflicts/Supersets for each patch are: Sub-Patch : 19769480 Bug Superset of 16870214 Super set bugs are: 16870214 [Apr 3, 2016 1:58:39 AM] OUI-67303: Patches [ 16870214 ] will be rolled back. So as we see the patch 16870214 was already applied on GI Home and again the patch is part of the jan2016 because of cumulative behavior, hence the patch will be rollback and again applied. So we no need to worry about it anymore. Patch installation In order to apply patch the OCM response file is mandatory, the file will be available under $ORACLE_HOME/Opatch/ocm/bin. If the file does not exist then we can again create using the file emocmrsp as below. The patch we must apply on both bodes and the procedure is same so below output belongs to only one node. $ cd /u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin oragrid@ORA-R2N1:+ASM1:/u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin > $ ./emocmrsp OCM Installation Response Generator 10.3.7.0.0 - Production Copyright (c) 2005, 2012, Oracle and/or its affiliates. All rights reserved. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y The OCM configuration response file (ocm.rsp) was successfully created. oragrid@ORA-R2N1:+ASM1:/u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin > $ ls -ltr total 20 -rw-r--r-- 1 oragrid oinstall 18 Jul 8 2008 dummy -rwxr-x--- 1 oragrid oinstall 9063 Apr 2 05:47 emocmrsp -rw-r--r-- 1 oragrid oinstall 624 Apr 3 04:24 ocm.rsp After the creation of ocm response file and we can start the patching, the important note is patching should not be run in parallel, of course there is no order on which node first we have to patch. Now the confusion is what about the MGMT database? Do we need to run the datapatch to load modified SQL after patching the GI home? The answer for the question is we no need to run the datapatch on MGMT database because GI(ASM) + MGMT all will be taken care. Again question is how to confirm whether the SQL patch was applied on the MGMT database? Yes we can refer the log files in order to gather the information. [root@ORA-R2N1 ~]# opatchauto apply /home/oragrid/patches/22191349 -oh /u01/app/oragrid/product/12.1.0/grid -ocmrf /u01/app/oragrid/product/12.1.0/grid/OPatch/ocm/bin/ocm.rsp OPatch Automation Tools Copyright (c)2014, Oracle Corporation. All rights reserved. OPatchauto Version : 12.1.0.1.10 OUI Version : 12.1.0.2.0 Running from : /u01/app/oragrid/product/12.1.0/grid opatchauto log file: /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatchauto/22191349/opatch_gi_2016-04-03_06-57-55_deploy.log Parameter Validation: Successful Configuration Validation: Successful Patch Location: /home/oragrid/patches/22191349 Grid Infrastructure Patch(es): 21436941 21948341 21948344 21948354 DB Patch(es): 21948344 21948354 Patch Validation: Successful User specified following Grid Infrastructure home: /u01/app/oragrid/product/12.1.0/grid Performing prepatch operations on CRS Home... Successful Applying patch(es) to "/u01/app/oragrid/product/12.1.0/grid" ... Patch "/home/oragrid/patches/22191349/21436941" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning. Patch "/home/oragrid/patches/22191349/21948341" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning. Patch "/home/oragrid/patches/22191349/21948344" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning. Patch "/home/oragrid/patches/22191349/21948354" applied to "/u01/app/oragrid/product/12.1.0/grid" with warning. Performing postpatch operations on CRS Home... Successful Apply Summary: opatchauto ran into some warnings during patch installation (Please see log file for details): GI Home: /u01/app/oragrid/product/12.1.0/grid: 21436941,21948341,21948344,21948354 opatchauto completed with warnings. [root@ORA-R2N1 ~]# From the above output we can see the patching was successful, but how to confirm whether the patching is also applicable on MGT database? Yes this information we can find from the log files of the location “$GRID_HOME/cfgtoollogs/crsconfig”. From the log file it is clear that sql patch also applied. 2016-04-03 06:54:37: Executing cmd: /u01/app/oragrid/product/12.1.0/grid/bin/srvctl status mgmtdb -S 1 2016-04-03 06:54:37: Command output: > #@=result[0]: dbunique_name={_mgmtdb} inst_name={-MGMTDB} node_name={ORA-R2N1} up={true} state_details={Open} internal_state={STABLE} >End Command output 2016-04-03 06:54:37: Mgmtdb is running on node: ORA-R2N1; local node: ORA-R2N1 2016-04-03 06:54:37: Mgmtdb is running on the local node 2016-04-03 06:54:37: Starting to patch Mgmt DB ... 2016-04-03 06:54:37: Invoking "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB" 2016-04-03 06:54:37: Running as user oragrid: /u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB 2016-04-03 06:54:37: Invoking "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB" as user "oragrid" 2016-04-03 06:54:37: Executing /bin/su oragrid -c "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB" 2016-04-03 06:54:37: Executing cmd: /bin/su oragrid -c "/u01/app/oragrid/product/12.1.0/grid/sqlpatch/sqlpatch -db -MGMTDB" 2016-04-03 06:55:48: Command output: > SQL Patching tool version 12.1.0.2.0 on Sun Apr 3 06:54:37 2016 > Copyright (c) 2015, Oracle. All rights reserved. > > Connecting to database...OK > Note: Datapatch will only apply or rollback SQL fixes for PDBs > that are in an open state, no patches will be applied to closed PDBs. > Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation > (Doc ID 1585822.1) > Determining current state...done > Adding patches to installation queue and performing prereq checks...done > Installation queue: > For the following PDBs: CDB$ROOT PDB$SEED ORA-R2N_CLUSTER > Nothing to roll back > The following patches will be applied: > 21948354 (Database Patch Set Update : 12.1.0.2.160119 (21948354)) > > Installing patches... > Patch installation complete. Total patches installed: 3 > > Validating logfiles...done > SQL Patching tool complete on Sun Apr 3 06:55:48 2016 >End Command output 2016-04-03 06:55:48: Successfully patched Mgmt D The patch is already applied, in order to know whether patch is applied on GI home, and we can run opatch lsinventory from the GI home to confirm. $ opatch lsinventory Oracle Interim Patch Installer version 12.1.0.1.10 Copyright (c) 2016, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oragrid/product/12.1.0/grid Central Inventory : /u01/app/oraInventory from : /u01/app/oragrid/product/12.1.0/grid/oraInst.loc OPatch version : 12.1.0.1.10 OUI version : 12.1.0.2.0 Log file location : /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatch/opatch2016-04-03_07-38-59AM_1.log Lsinventory Output file location : /u01/app/oragrid/product/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2016-04-03_07-38-59AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: 10.128.206.11 ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Grid Infrastructure 12c 12.1.0.2.0 There are 1 products installed in this Oracle Home. Interim patches (4) : Patch 21948354 : applied on Sun Apr 03 06:49:12 EDT 2016 Unique Patch ID: 19553095 Patch description: "Database Patch Set Update : 12.1.0.2.160119 (21948354)" Created on 20 Dec 2015, 23:39:33 hrs PST8PDT Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)" Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)" Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)" Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)" Bugs fixed: 19189525, 19075256, 19141838, 19865345, 19791273, 19280225, 18845653 . . . . . . Patch 21436941 : applied on Sun Apr 03 06:47:51 EDT 2016 Unique Patch ID: 19245012 Patch description: "WLM Patch Set Update: 12.1.0.2.5 (21436941)" Created on 13 Aug 2015, 04:00:40 hrs PST8PDT Bugs fixed: 20075154, 20641027, 20548410, 19016964, 19582630 Patch level status of Cluster nodes : Patching Level Nodes -------------- ----- 2278979115 ORA-R2N2,ORA-R2N1 ----------------------------------------------------------- OPatch succeeded. oragrid@ORA-R2N1:+ASM1:/home/oragrid > $ By thus we have completed all the steps related to GI home and we can start patching the database home. Patching Database Home As said earlier, in fact patching on database home will take care by opatchauto from root user but that is applicable when the GI and database homes are in same version, release, patchset. Where there is difference in any of them we must consider to apply patch on GI home alone and later RDBMS home later. The prerequisites are same as GI home like Opatch version should met the requirements as per the readme file or it can be higher version. If opatch version is low then it can be downloaded from the patch number 6880880. $ export PATH=$ORACLE_HOME/OPatch:$PATH $ opatch version OPatch Version: 12.1.0.1.10 OPatch succeeded. $ Patch Conflict Detection Unzip the downloaded patch and prior to applying the patch, we must check whether the new patch we are going to apply can be able to patch or not. $ cd /home/oracle/patches/db_patch/21951844 $ opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.1.0.1.10 Copyright (c) 2016, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/12.1.0/db Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/db/oraInst.loc OPatch version : 12.1.0.1.10 OUI version : 12.1.0.1.0 Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-02_08-17-27AM_1.log Invoking prereq "checkconflictagainstohwithdetail" ZOP-40: The patch(es) has conflicts with other patches installed in the Oracle Home (or) among themselves. Prereq "checkConflictAgainstOHWithDetail" failed. Summary of Conflict Analysis: There are no patches that can be applied now. Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches : 16870214, 21951844 Whole composite patch Conflicts/Supersets are: Composite Patch : 21951844 Conflict with 16870214 Detail Conflicts/Supersets for each patch are: Sub-Patch : 19769486 Conflict with 16870214 Conflict details: /u01/app/oracle/product/12.1.0/db/lib/libserver12.a:ksp.o /u01/app/oracle/product/12.1.0/db/lib/libserver12.a:ksfd.o OPatch succeeded. $ We can see the conflict with the patch on the current RDBMS home, In order to apply patch we have to few things 1) rollback the patch 16870214 2) apply the PSU patch 3) apply overlay patch of 16870214. The patching we have to start once the overlay patch is available to download. Rollback of the patch we must do from both the nodes. Initially we will stop the oracle database prior to patching. $ srvctl stop database -d hadb oracle@ORA-R2N1:hadb1:/home/oracle > $ srvctl status database -d hadb Instance hadb1 is not running on node ORA-R2N1 Instance hadb2 is not running on node ORA-R2N2 $ export PATH=$ORACLE_HOME/OPatch:$PATH $ opatch rollback -id 16870214 -local Oracle Interim Patch Installer version 12.1.0.1.10 Copyright (c) 2016, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/db Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/db/oraInst.loc OPatch version : 12.1.0.1.10 OUI version : 12.1.0.1.0 Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_05_18_50/rollback2016-04-03_05-18-50AM_1.log RollbackSession rolling back interim patch '16870214' from OH '/u01/app/oracle/product/12.1.0/db' Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.1.0/db') Is the local system ready for patching? [y|n] y User Responded with: Y Patching component oracle.rdbms, 12.1.0.1.0... RollbackSession removing interim patch '16870214' from inventory Log file location: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_05_18_50/rollback2016-04-03_05-18-50AM_1.log OPatch succeeded. $ The above rollback procedure we have to run from 2 nd node as well, After that we can start apply January 2016 PSU patch after performing conflict detection again whether the patch will succeed or not $ cd /home/oracle/patches/db_patch/21951844 oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/21951844 > $ opatch apply -local Oracle Interim Patch Installer version 12.1.0.1.10 Copyright (c) 2016, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/db Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/db/oraInst.loc OPatch version : 12.1.0.1.10 OUI version : 12.1.0.1.0 Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-03_07-52-30AM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 19769486 20299016 20831107 21352619 21951844 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.1.0/db') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying sub-patch '19769486' to OH '/u01/app/oracle/product/12.1.0/db' ApplySession: Optional component(s) [ oracle.network.cman, 12.1.0.1.0 ] , [ oracle.oraolap, 12.1.0.1.0 ] not present in the Oracle Home or a higher version is found. Patching component oracle.network.rsf, 12.1.0.1.0... Patching component oracle.ldap.rsf, 12.1.0.1.0... . . . . . . Patching component oracle.xdk, 12.1.0.1.0... Patching component oracle.rdbms, 12.1.0.1.0... Composite patch 21951844 successfully applied. Log file location: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-03_07-52-30AM_1.log OPatch succeeded. oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/21951844 > Overlay for Rollback Patch Before applying the Jan2016 PSU we have performed roll back of conflict patch and now we can apply the overlay patch based on the database version and release we are using. The overlay patch we can download from MOS and based on that we have to search with the same patch number. In my case the overlay patch can be downloaded from http://updates.oracle.com/download/16870214.html $ cd /home/oracle/patches/db_patch/overlay/16870214 oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/overlay/16870214 > $ opatch apply -local Oracle Interim Patch Installer version 12.1.0.1.10 Copyright (c) 2016, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/db Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/db/oraInst.loc OPatch version : 12.1.0.1.10 OUI version : 12.1.0.1.0 Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_07_55_37/apply2016-04-03_07-55-37AM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 16870214 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: You have not provided an email address for notification of security issues. Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/u01/app/oracle/product/12.1.0/db') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '16870214' to OH '/u01/app/oracle/product/12.1.0/db' Patching component oracle.rdbms, 12.1.0.1.0... Patch 16870214 successfully applied. Log file location: /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/16870214_Apr_03_2016_07_55_37/apply2016-04-03_07-55-37AM_1.log OPatch succeeded. oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/overlay/16870214 > Check for the applied patches using lsinventory $ opatch lsinventory Oracle Interim Patch Installer version 12.1.0.1.10 Copyright (c) 2016, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/12.1.0/db Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/12.1.0/db/oraInst.loc OPatch version : 12.1.0.1.10 OUI version : 12.1.0.1.0 Log file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/opatch2016-04-03_05-30-55AM_1.log Lsinventory Output file location : /u01/app/oracle/product/12.1.0/db/cfgtoollogs/opatch/lsinv/lsinventory2016-04-03_05-30-55AM.txt -------------------------------------------------------------------- Local Machine Information:: Hostname: ORA-R2N1.oracle-ckpt.com ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Database 12c 12.1.0.1.0 There are 1 products installed in this Oracle Home. Interim patches (2) : Patch 16870214 : applied on Sun Apr 03 05:29:16 EDT 2016 Unique Patch ID: 18465450 Created on 19 Jan 2015, 22:43:42 hrs PST8PDT Bugs fixed: 16870214 This patch overlays patches: 19769486 This patch needs patches: 19769486 as prerequisites Patch 21951844 : applied on Sun Apr 03 05:22:43 EDT 2016 Unique Patch ID: 19567210 Patch description: "Database Patch Set Update : 12.1.0.1.160119 (21951844)" Created on 17 Nov 2015, 01:48:00 hrs PST8PDT Sub-patch 21352619; "Database Patch Set Update : 12.1.0.1.9 (21352619)" Sub-patch 20831107; "Database Patch Set Update : 12.1.0.1.8 (20831107)" Sub-patch 20299016; "Database Patch Set Update : 12.1.0.1.7 (20299016)" Sub-patch 19769486; "Database Patch Set Update : 12.1.0.1.6 (19769486)" . . . . . . Rac system comprising of multiple nodes Local node = ORA-R2N1 Remote node = ORA-R2N2 -------------------------------------------------------------------------------- OPatch succeeded. $ Loading Modified SQL After the all the patches applied on RDBMS home, we can run the datapatch from the database level as below after starting any one node. SQL> startup ORACLE instance started. Total System Global Area 2.6724E+10 bytes Fixed Size 7584272 bytes Variable Size 1.4764E+10 bytes Database Buffers 1.1878E+10 bytes Redo Buffers 73891840 bytes Database mounted. Database opened. SQL> exit Disconnected from Oracle Database 12c Release 12.1.0.1.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options oracle@ORA-R2N1:hadb1:/home/oracle/patches/db_patch/overlay/16870214 > $ cd $ORACLE_HOME/OPatch oracle@ORA-R2N1:hadb1:/u01/app/oracle/product/12.1.0/db/OPatch > $ ./datapatch -verbose SQL Patching tool version 12.1.0.1.0 on Sun Apr 3 07:57:35 2016 Copyright (c) 2014, Oracle. All rights reserved. Connecting to database...OK Determining current state... Currently installed SQL Patches: Currently installed C Patches: 19769486,20299016,20831107,21352619,21951844 Adding patches to installation queue and performing prereq checks... Installation queue: Nothing to roll back The following patches will be applied: 19769486,20299016,20831107,21352619,21951844 Installing patches... Patch installation complete. Total patches installed: 5 Validating logfiles... Patch 19769486 apply: SUCCESS logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/19769486/19769486_apply_hadb_2016Apr03_07_58_41.log (no errors) catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors) catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors) Patch 20299016 apply: SUCCESS logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/20299016/20299016_apply_hadb_2016Apr03_08_00_15.log (no errors) catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors) catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors) Patch 20831107 apply: SUCCESS logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/20831107/20831107_apply_hadb_2016Apr03_08_00_15.log (no errors) catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors) catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors) Patch 21352619 apply: SUCCESS logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/21352619/21352619_apply_hadb_2016Apr03_08_00_15.log (no errors) catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors) catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors) Patch 21951844 apply: SUCCESS logfile: /u01/app/oracle/product/12.1.0/db/sqlpatch/21951844/21951844_apply_hadb_2016Apr03_08_00_16.log (no errors) catbundle generate logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_GENERATE_2016Apr03_08_00_16.log (no errors) catbundle apply logfile: /u01/app/oracle/cfgtoollogs/catbundle/catbundle_PSU_hadb_hadb_APPLY_2016Apr03_08_00_16.log (no errors) SQL Patching tool complete on Sun Apr 3 08:00:16 2016 oracle@ORA-R2N1:hadb1:/u01/app/oracle/product/12.1.0/db/OPatch > $ From the log files, we can see there are no errors seen while datapatch is being applied, So we are done with all the steps related to Database home. Ensure CRS and Database all services are up Now start all the services and enable the crontab jobs and backup the database if necessary as required. [root@ORA-R2N1 patches]# crsctl stop cluster -all CRS-2673: Attempting to stop 'ora.crsd' on 'ORA-R2N1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.REDODG4.dg' on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.DATADG1.dg' on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.FRADG2.dg' on 'ORA-R2N1' CRS-2673: Attempting to stop 'ora.REDODG3.dg' on 'ORA-R2N1' . . . . . . CRS-2673: Attempting to stop 'ora.cssd' on 'ORA-R2N2' CRS-2677: Stop of 'ora.cssd' on 'ORA-R2N1' succeeded CRS-2677: Stop of 'ora.cssd' on 'ORA-R2N2' succeeded [root@ORA-R2N1 patches]# crsctl start cluster -all CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ORA-R2N1' CRS-2672: Attempting to start 'ora.evmd' on 'ORA-R2N2' CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ORA-R2N2' CRS-2672: Attempting to start 'ora.evmd' on 'ORA-R2N1' CRS-2676: Start of 'ora.cssdmonitor' on 'ORA-R2N1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'ORA-R2N1' CRS-2672: Attempting to start 'ora.diskmon' on 'ORA-R2N1 . . . . . . CRS-2676: Start of 'ora.crsd' on 'ORA-R2N2' succeeded [root@ORA-R2N1 patches] [root@ORA-R2N1 patches]# ps -ef|grep pmon oragrid 7388 1 0 08:02 ? 00:00:00 asm_pmon_+ASM1 oracle 9315 1 0 08:03 ? 00:00:00 ora_pmon_hadb1 oragrid 12516 1 0 08:03 ? 00:00:00 mdb_pmon_-MGMTDB root 14623 40282 0 08:04 pts/1 00:00:00 grep --color=auto pmon [root@ORA-R2N1 patches]# $ srvctl status database -d hadb Instance hadb1 is running on node ORA-R2N1 Instance hadb2 is running on node ORA-R2N2 oracle@ORA-R2N1:hadb1:/home/oracle > $ Summary We have seen how to perform PSU patch when GI 12.1.0.2 when the MGMT database is running from GI home with step by step procedure along with the brief introduction of GIMR and also we have seen how to manage when the version/release/patchset is different to database home so on, We also have seen how to patching the database home and how to deal in case of conflicts. Reference OPatch patch of version 12.1.0.1.10 for Oracle software releases 12.1.0.x (NOV 2015) Patch 21951844 - Database Patch Set Update 12.1.0.1.160119 (Jan2016) Patch 22191349 - Oracle Grid Infrastructure Patch Set Update 12.1.0.2.160119 (Jan2016) https://blogs.oracle.com/UPGRADE/entry/grid_infrastructure_management_repository_gimr1
↧