Quantcast
Channel: Oracle
Viewing all articles
Browse latest Browse all 1814

Wiki Page: Exadata Write-back Flash Cache

$
0
0
About Exadata Write-Back Flash Cache By defaults, Exadata Database Machine operates in WriteThrough mode, in which it only caches the read IOs to the PCI flash. Write-Back Flash Cache (WBFC) provides the ability to cache write IOs directly to the PCI flash in addition to the read IOs. Write operations serviced by flash are known as Write-Back Flash Cache. WBFC significantly improves the performance of write operations. Advantages of WBFC: · Caches both read and write IOs to the PCI flash · Improved performs for write operations · Improved query response time and throughput. · Application with intensive write operations and waits for “free buffer waits” can benefit using WBFC · Exadata WBFC is persistent across storage cell reboots. · You can Enable or Disable WBFC at cell level. Enable & Disable Write-Back Flash Cache – Using Automation Script WBFC can be Enabled or Disabled in 2 ways. - Manually using “cellcli” commands –or– - WBFC Automation Script Automation script automates all the steps that were previous done manually. It supports both rolling and non-rolling. In rolling method, this script executes 9 steps on each cell, one cell at a time, several steps will be skipped if the storage software is 11.2.3.3.1 or higher. Following are the steps that will be performed: - Ensure that grid disk status is not OFFLINE - Drop Flash Cache - Inactivate grid disks - Shutdown CELLSRV - Set the Flash Cache Mode to Write-Back - Restart CELLSRV - Activate grid disks - Re-create Flash Cache - Verify Flash Cache Mode - Wait grid disks to synchronize and be ONLINE For non-rolling, this script executes 8 steps on each cell in parallel. Several steps will be skipped if the storage software is 11.2.3.3.1 or higher. Note: Exadata X5-2 Extreme Flash comes with WBFC enabled. In this article we will describe the steps to Enable & Disable Write-Back Flash Cache mode using WBFC automation script in different ways. · Enable WBFC In Rolling Fashion · Enable WBFC In Non-Rolling Fashion · Disable WBFC In Non-Rolling Fashion Environment Details Exadata Model X5-2 Half Rack HC 4TB Exadata Components Storage Cell (7), Compute node (4) & Infiniband Switch (2) Exadata Storage cells DBM01CEL01 – DBM01CEL07 Exadata Compute nodes DBM01DB01 – DBM01DB04 Exadata Software Version 12.1.2.1.1 Exadata DB Version 11.2.0.4 BP15 Installation procedure 1. Prerequisites o WBFC automation script must be run as Root user. o “cell_group” file containing list of cell nodes, one per line. o “dbs_group” file containing list of DB nodes in the cluster, one per line. o Root User equivalence must be setup between all the compute nodes and storage cells. #dcli -g cell_group -l root -k o Exadata storage software version 11.2.3.2.1 is the minimum version required to use write back flash cache. 2. Supported Storage cell software versions o 11.2.3.2.1 and above o 11.2.3.3.1 o 12.1.1.1.1 o 12.1.2.1.0 o 12.1.2.1.1 3. Grid Infrastructure versions o 11.2.0.3.9 and above o 12.1.0.1 4. Software Download and Copy it to the Server Download the latest WBFC automation script from MOS note ID 1500257.1 Copy the WBFC automation script to the compute node 1 into a directory using WinSCP or similar utility. In this example I copied the script to DBM01DB01 under /u01/app/oracle/stage directory. [root@dbm01db01 ~]# cd /u01/app/oracle/stage [root@dbm01db01 stage]# ls -ltr -rw-r--r-- 1 root root 27641 Mar 12 03:52 setWBFC_10016_20140716.zip 5. Configuration steps COMMON STEPS i. Unzip the WBFC automation script [root@dbm01db01 stage]# unzip setWBFC_10016_20140716.zip Archive: setWBFC_10016_20140716.zip inflating: setWBFC.sh inflating: wbfc_FLUSH.sh inflating: WBFC_README.txt ii. Create a directory for storing WBFC log files. [root@dbm01db01 stage]# mkdir setWBFC_log iii. Execute the following command to display the options that can be used with setWBFC.sh [root@dbm01db01 stage]# ./setWBFC.sh setWBFC Version: 1.0.0.1.6.20140716 Usage: ./setWBFC.sh -g cell_group_file [-d dbs_group_file ] [ -h ] [ -i ] [ -l log_directory ] [ -m WriteBack | WriteThrough ] [ -o rolling | non-rolling ] [ -p ] [ -s step_number ] [ -t time_out_seconds ] [ -x trace_level ] [ -v ] -g file file that lists cell host names, one per line -d file file that lists the database host names, one per line. Required for non-rolling. -h help, print this information -i run in interactive mode -l log directory directory path for log files -m FC_mode flashcache mode: WriteBack | WriteThrough -o exec_mode execution mode: rolling | non-rolling (default) -p perform a precheck only -s step # (*) specify step number to restart at -t timeout sec specify in seconds the amount of time to wait for griddisks to come ONLINE - range: [600 - 43200] Default: 21600 (6 hours) -x trace level # specify trace level for further diagnostics -v show version (*) -- Option not yet implemented. iv. Verify current flash cache policy. [root@dbm01db01 stage]# dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode" dbm01cel01: WriteThrough dbm01cel02: WriteThrough dbm01cel03: WriteThrough dbm01cel04: WriteThrough dbm01cel05: WriteThrough dbm01cel06: WriteThrough dbm01cel07: WriteThrough Currently the flash cache policy is set to “WriteThrough” ENABLE WBFC IN ROLLING FASHION For rolling, this script executes 9 steps on each cell, one cell at a time, one after another. Several steps will be skipped if the storage software is 11.2.3.3.1 or higher. i. Run the following command to perform a prerequisite check. It is recommended to perform pre-req checks before running the script in update mode. Note: If you do not specify the -l option to specify where this directory should be, it will create a subdirectory called WBFC_log under the install directory [root@dbm01db01 stage]# ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteBack -o rolling -p ./setWBFC.sh: Using log directory 'setWBFC_log' ./setWBFC.sh: Log File 'setWBFC_log/setWBFC_62566_2015-05-11-09:11:01.log' created successfully 2015-05-11 09:11:01 Starting ./setWBFC.sh on dbm01db01 Version: 1.0.0.1.6.20140716 Command line options used: -g /root/cell_group -d /root/dbs_group -i (interractive) -o rolling -m WriteBack -p (Perform pre-req checks only) -t 21600 -x 0 2015-05-11 09:11:01 Performing pre-req checks..... 2015-05-11 09:11:01 Creating baseline inventory for griddisks 2015-05-11 09:11:02 Creating baseline inventory for flashdisks 2015-05-11 09:11:03 Creating baseline inventory for flashsize 2015-05-11 09:11:04 dcli present and in PATH. [PASSED] 2015-05-11 09:11:04 Checking cell nodes are valid storage servers... 2015-05-11 09:11:04 All cells are valid Exadata storage cells. 2015-05-11 09:11:04 Checking Exadata Storage Software Versions... 2015-05-11 09:11:08 Software versions of the following cells: dbm01cel01: 12.1.2.1.1.150316.2 [PASSED] dbm01cel02: 12.1.2.1.1.150316.2 [PASSED] dbm01cel03: 12.1.2.1.1.150316.2 [PASSED] dbm01cel04: 12.1.2.1.1.150316.2 [PASSED] dbm01cel05: 12.1.2.1.1.150316.2 [PASSED] dbm01cel06: 12.1.2.1.1.150316.2 [PASSED] dbm01cel07: 12.1.2.1.1.150316.2 [PASSED] 2015-05-11 09:11:08 Checking Grid Infrastructure Software Version... 2015-05-11 09:11:10 Grid Infrastructure version: 11.2.0.4.15 [PASSED] 2015-05-11 09:11:10 Checking for active ASM operations.... 2015-05-11 09:11:10 Check for no active ASM operations: [PASSED] 2015-05-11 09:11:10 Checking griddisk status across all cells.... 2015-05-11 09:11:15 All griddisks across all cells have asmdeactivationoutcome = Yes All griddisks across all cells are ONLINE Griddisk checks: [PASSED] 2015-05-11 09:11:15 Checking flash cache status..... 2015-05-11 09:11:16 Flashcache status normal: [PASSED] 2015-05-11 09:11:16 Checking that all FlashDisks are present... 2015-05-11 09:11:17 FlashDisk validation: [PASSED] 2015-05-11 09:11:17 Checking current flash cache mode..... 2015-05-11 09:11:18 Flashcache not already in target mode: [PASSED] 2015-05-11 09:11:18 All pre-req checks completed: [PASSED] 2015-05-11 09:11:19 dbm01cel01: flashcache size: 5.82122802734375T dbm01cel02: flashcache size: 5.82122802734375T dbm01cel03: flashcache size: 5.82122802734375T dbm01cel04: flashcache size: 5.82122802734375T dbm01cel05: flashcache size: 5.82122802734375T dbm01cel06: flashcache size: 5.82122802734375T dbm01cel07: flashcache size: 5.82122802734375T There are 7 storage cells to process. Troubleshoot if there are issues. ii. Now ENABLE the WBFC mode using the following command in ROLLING Mode. [root@dbm01db01 stage]# ./setWBFC.sh -g ~/cell_group -i -l setWBFC_log -m WriteBack -o rolling ./setWBFC.sh: Using log directory 'setWBFC_log' ./setWBFC.sh: Log File 'setWBFC_log/setWBFC_67992_2015-05-11-09:11:34.log' created successfully 2015-05-11 09:11:34 Starting ./setWBFC.sh on dbm01db01 Version: 1.0.0.1.6.20140716 Command line options used: -g /root/cell_group -d /root/dbs_group -i (interractive) -o rolling -m WriteBack -t 21600 -x 0 2015-05-11 09:11:34 Performing pre-req checks..... 2015-05-11 09:11:34 Creating baseline inventory for griddisks 2015-05-11 09:11:35 Creating baseline inventory for flashdisks 2015-05-11 09:11:36 Creating baseline inventory for flashsize 2015-05-11 09:11:36 dcli present and in PATH. [PASSED] 2015-05-11 09:11:36 Checking cell nodes are valid storage servers... 2015-05-11 09:11:37 All cells are valid Exadata storage cells. 2015-05-11 09:11:37 Checking Exadata Storage Software Versions... 2015-05-11 09:11:41 Software versions of the following cells: dbm01cel01: 12.1.2.1.1.150316.2 [PASSED] dbm01cel02: 12.1.2.1.1.150316.2 [PASSED] dbm01cel03: 12.1.2.1.1.150316.2 [PASSED] dbm01cel04: 12.1.2.1.1.150316.2 [PASSED] dbm01cel05: 12.1.2.1.1.150316.2 [PASSED] dbm01cel06: 12.1.2.1.1.150316.2 [PASSED] dbm01cel07: 12.1.2.1.1.150316.2 [PASSED] 2015-05-11 09:11:41 Checking Grid Infrastructure Software Version... 2015-05-11 09:11:42 Grid Infrastructure version: 11.2.0.4.15 [PASSED] 2015-05-11 09:11:42 Checking for active ASM operations.... 2015-05-11 09:11:43 Check for no active ASM operations: [PASSED] 2015-05-11 09:11:43 Checking griddisk status across all cells.... 2015-05-11 09:11:45 All griddisks across all cells have asmdeactivationoutcome = Yes All griddisks across all cells are ONLINE Griddisk checks: [PASSED] 2015-05-11 09:11:45 Checking flash cache status..... 2015-05-11 09:11:46 Flashcache status normal: [PASSED] 2015-05-11 09:11:46 Checking that all FlashDisks are present... 2015-05-11 09:11:48 FlashDisk validation: [PASSED] 2015-05-11 09:11:48 Checking current flash cache mode..... 2015-05-11 09:11:49 Flashcache not already in target mode: [PASSED] 2015-05-11 09:11:49 All pre-req checks completed: [PASSED] 2015-05-11 09:11:50 dbm01cel01: flashcache size: 5.82122802734375T dbm01cel02: flashcache size: 5.82122802734375T dbm01cel03: flashcache size: 5.82122802734375T dbm01cel04: flashcache size: 5.82122802734375T dbm01cel05: flashcache size: 5.82122802734375T dbm01cel06: flashcache size: 5.82122802734375T dbm01cel07: flashcache size: 5.82122802734375T There are 7 storage cells to process. All pre-req checks are complete. Ready to proceed with changing the flash cache mode to WriteBack. OK to continue? [Y|y|N|n] :y Continuing.... 2015-05-11 09:11:58 Changing flash cache to WriteBack ROLLING.... Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel01. OK to continue? [Y|y|N|n] :y 2015-05-11 09:12:01 STEP 0: Checking gridisk status on cell: dbm01cel01 2015-05-11 09:12:03 STEP 0 completed successfully on cell: dbm01cel01 2015-05-11 09:12:06 STEP 1: Dropping flashcache on cell: dbm01cel01 2015-05-11 09:12:42 STEP 1: Completed sucessfully on cell: dbm01cel01 2015-05-11 09:12:45 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:12:45 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:12:45 STEP 4: Set the flashCachMode on cell: dbm01cel01 2015-05-11 09:12:45 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:12:46 STEP 4: Completed successfully on cell: dbm01cel01 2015-05-11 09:12:49 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:12:49 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:12:49 STEP 7: Creating flashcache on cell: dbm01cel01 2015-05-11 09:13:39 STEP 7: Completed sucessfully on cell: dbm01cel01 2015-05-11 09:13:42 STEP 8: Verifying flashCacheMode on cell: dbm01cel01 2015-05-11 09:13:43 STEP 8: Completed sucessfully on cell: dbm01cel01 Flash Cache mode is now WriteBack 2015-05-11 09:13:46 Skipping STEP 9: Waiting for grid disks to sync not required. Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel02. OK to continue? [Y|y|N|n] :y 2015-05-11 09:13:53 STEP 0: Checking gridisk status on cell: dbm01cel02 2015-05-11 09:13:54 STEP 0 completed successfully on cell: dbm01cel02 2015-05-11 09:13:57 STEP 1: Dropping flashcache on cell: dbm01cel02 2015-05-11 09:14:35 STEP 1: Completed sucessfully on cell: dbm01cel02 2015-05-11 09:14:38 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:14:38 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:14:38 STEP 4: Set the flashCachMode on cell: dbm01cel02 2015-05-11 09:14:38 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:14:39 STEP 4: Completed successfully on cell: dbm01cel02 2015-05-11 09:14:42 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:14:42 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:14:42 STEP 7: Creating flashcache on cell: dbm01cel02 2015-05-11 09:15:29 STEP 7: Completed sucessfully on cell: dbm01cel02 2015-05-11 09:15:32 STEP 8: Verifying flashCacheMode on cell: dbm01cel02 2015-05-11 09:15:33 STEP 8: Completed sucessfully on cell: dbm01cel02 Flash Cache mode is now WriteBack 2015-05-11 09:15:36 Skipping STEP 9: Waiting for grid disks to sync not required. Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel03. OK to continue? [Y|y|N|n] :y 2015-05-11 09:15:42 STEP 0: Checking gridisk status on cell: dbm01cel03 2015-05-11 09:15:44 STEP 0 completed successfully on cell: dbm01cel03 2015-05-11 09:15:47 STEP 1: Dropping flashcache on cell: dbm01cel03 2015-05-11 09:16:20 STEP 1: Completed sucessfully on cell: dbm01cel03 2015-05-11 09:16:23 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:16:23 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:16:23 STEP 4: Set the flashCachMode on cell: dbm01cel03 2015-05-11 09:16:23 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:16:23 STEP 4: Completed successfully on cell: dbm01cel03 2015-05-11 09:16:26 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:16:26 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:16:26 STEP 7: Creating flashcache on cell: dbm01cel03 2015-05-11 09:17:14 STEP 7: Completed sucessfully on cell: dbm01cel03 2015-05-11 09:17:17 STEP 8: Verifying flashCacheMode on cell: dbm01cel03 2015-05-11 09:17:18 STEP 8: Completed sucessfully on cell: dbm01cel03 Flash Cache mode is now WriteBack 2015-05-11 09:17:21 Skipping STEP 9: Waiting for grid disks to sync not required. Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel04. OK to continue? [Y|y|N|n] :y 2015-05-11 09:17:37 STEP 0: Checking gridisk status on cell: dbm01cel04 2015-05-11 09:17:38 STEP 0 completed successfully on cell: dbm01cel04 2015-05-11 09:17:41 STEP 1: Dropping flashcache on cell: dbm01cel04 2015-05-11 09:18:21 STEP 1: Completed sucessfully on cell: dbm01cel04 2015-05-11 09:18:24 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:18:24 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:18:24 STEP 4: Set the flashCachMode on cell: dbm01cel04 2015-05-11 09:18:24 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:18:25 STEP 4: Completed successfully on cell: dbm01cel04 2015-05-11 09:18:28 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:18:28 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:18:28 STEP 7: Creating flashcache on cell: dbm01cel04 2015-05-11 09:19:17 STEP 7: Completed sucessfully on cell: dbm01cel04 2015-05-11 09:19:20 STEP 8: Verifying flashCacheMode on cell: dbm01cel04 2015-05-11 09:19:21 STEP 8: Completed sucessfully on cell: dbm01cel04 Flash Cache mode is now WriteBack 2015-05-11 09:19:24 Skipping STEP 9: Waiting for grid disks to sync not required. Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel05. OK to continue? [Y|y|N|n] :y 2015-05-11 09:20:17 STEP 0: Checking gridisk status on cell: dbm01cel05 2015-05-11 09:20:19 STEP 0 completed successfully on cell: dbm01cel05 2015-05-11 09:20:22 STEP 1: Dropping flashcache on cell: dbm01cel05 2015-05-11 09:21:03 STEP 1: Completed sucessfully on cell: dbm01cel05 2015-05-11 09:21:06 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:21:06 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:21:06 STEP 4: Set the flashCachMode on cell: dbm01cel05 2015-05-11 09:21:06 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:21:06 STEP 4: Completed successfully on cell: dbm01cel05 2015-05-11 09:21:09 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:21:09 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:21:09 STEP 7: Creating flashcache on cell: dbm01cel05 2015-05-11 09:21:58 STEP 7: Completed sucessfully on cell: dbm01cel05 2015-05-11 09:22:01 STEP 8: Verifying flashCacheMode on cell: dbm01cel05 2015-05-11 09:22:02 STEP 8: Completed sucessfully on cell: dbm01cel05 Flash Cache mode is now WriteBack 2015-05-11 09:22:05 Skipping STEP 9: Waiting for grid disks to sync not required. Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel06. OK to continue? [Y|y|N|n] :y 2015-05-11 09:23:22 STEP 0: Checking gridisk status on cell: dbm01cel06 2015-05-11 09:23:24 STEP 0 completed successfully on cell: dbm01cel06 2015-05-11 09:23:27 STEP 1: Dropping flashcache on cell: dbm01cel06 2015-05-11 09:24:07 STEP 1: Completed sucessfully on cell: dbm01cel06 2015-05-11 09:24:10 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:24:10 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:24:10 STEP 4: Set the flashCachMode on cell: dbm01cel06 2015-05-11 09:24:10 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:24:11 STEP 4: Completed successfully on cell: dbm01cel06 2015-05-11 09:24:14 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:24:14 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:24:14 STEP 7: Creating flashcache on cell: dbm01cel06 2015-05-11 09:25:02 STEP 7: Completed sucessfully on cell: dbm01cel06 2015-05-11 09:25:05 STEP 8: Verifying flashCacheMode on cell: dbm01cel06 2015-05-11 09:25:06 STEP 8: Completed sucessfully on cell: dbm01cel06 Flash Cache mode is now WriteBack 2015-05-11 09:25:09 Skipping STEP 9: Waiting for grid disks to sync not required. Ready to proceed with changing the flash cache mode to WriteBack on cell: dbm01cel07. OK to continue? [Y|y|N|n] :y 2015-05-11 09:25:13 STEP 0: Checking gridisk status on cell: dbm01cel07 2015-05-11 09:25:15 STEP 0 completed successfully on cell: dbm01cel07 2015-05-11 09:25:18 STEP 1: Dropping flashcache on cell: dbm01cel07 2015-05-11 09:25:50 STEP 1: Completed sucessfully on cell: dbm01cel07 2015-05-11 09:25:53 Skipping STEP 2: Inactivating grid disks not required. 2015-05-11 09:25:53 Skipping STEP 3: Shutdown of cellsrv not required. 2015-05-11 09:25:53 STEP 4: Set the flashCachMode on cell: dbm01cel07 2015-05-11 09:25:53 STEP 4: Setting flashCacheMode to WriteBack 2015-05-11 09:26:00 STEP 4: Completed successfully on cell: dbm01cel07 2015-05-11 09:26:03 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:26:03 Skipping STEP 6: Acativating grid disks not required. 2015-05-11 09:26:03 STEP 7: Creating flashcache on cell: dbm01cel07 2015-05-11 09:26:51 STEP 7: Completed sucessfully on cell: dbm01cel07 2015-05-11 09:26:54 STEP 8: Verifying flashCacheMode on cell: dbm01cel07 2015-05-11 09:26:55 STEP 8: Completed sucessfully on cell: dbm01cel07 Flash Cache mode is now WriteBack 2015-05-11 09:26:58 Skipping STEP 9: Waiting for grid disks to sync not required. 2015-05-11 09:26:58 Validating inventory for griddisks 2015-05-11 09:26:59 Validation of griddisk: [PASSED] 2015-05-11 09:26:59 Validating inventory for flashdisks 2015-05-11 09:27:02 Validation of flashdisk: [PASSED] 2015-05-11 09:27:02 Validating inventory for flashsize 2015-05-11 09:27:03 Validation of flashsize: [PASSED] 2015-05-11 09:27:03 Setting flash cache to WriteBack completed successfully. ENABLE WBFC IN NON-ROLLING FASHION For non-rolling, this script performs 8 steps on each cell. Each step is executed in parallel on all cells. Several steps will be skipped if the storage software is 11.2.3.3.1 or higher. i. Run the following command to perform a prerequisite check. It is recommended to perform pre-req checks before running the script in update mode. Note: If you do not specify the -l option to specify where this directory should be, it will create a subdirectory called WBFC_log under the install directory [root@dbm01db01 stage]# ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteBack -o non-rolling -p ./setWBFC.sh: Using log directory 'setWBFC_log' ./setWBFC.sh: Log File 'setWBFC_log/setWBFC_46237_2015-03-12-03:59:04.log' created successfully 2015-03-12 03:59:04 Starting ./setWBFC.sh on dbm01db01 Version: 1.0.0.1.6.20140716 Command line options used: -g /root/cell_group -d /root/dbs_group -i (interractive) -o non-rolling -m WriteBack -p (Perform pre-req checks only) -t 21600 -x 0 2015-03-12 03:59:04 Performing pre-req checks..... 2015-03-12 03:59:04 Creating baseline inventory for griddisks 2015-03-12 03:59:05 Creating baseline inventory for flashdisks 2015-03-12 03:59:06 Creating baseline inventory for flashsize 2015-03-12 03:59:07 dcli present and in PATH. [PASSED] 2015-03-12 03:59:07 Checking cell nodes are valid storage servers... 2015-03-12 03:59:07 All cells are valid Exadata storage cells. 2015-03-12 03:59:07 Checking Exadata Storage Software Versions... 2015-03-12 03:59:11 Software versions of the following cells: dbm01cel01: 12.1.2.1.0.141206.1 [PASSED] dbm01cel02: 12.1.2.1.0.141206.1 [PASSED] dbm01cel03: 12.1.2.1.0.141206.1 [PASSED] dbm01cel04: 12.1.2.1.0.141206.1 [PASSED] dbm01cel05: 12.1.2.1.0.141206.1 [PASSED] dbm01cel06: 12.1.2.1.0.141206.1 [PASSED] dbm01cel07: 12.1.2.1.0.141206.1 [PASSED] 2015-03-12 03:59:11 Checking Grid Infrastructure Software Version... 2015-03-12 03:59:12 Grid Infrastructure version: 11.2.0.4.15 [PASSED] 2015-03-12 03:59:12 Checking for active ASM operations.... 2015-03-12 03:59:13 Check for no active ASM operations: [PASSED] 2015-03-12 03:59:13 Checking griddisk status across all cells.... 2015-03-12 03:59:24 All griddisks across all cells have asmdeactivationoutcome = Yes All griddisks across all cells are ONLINE Griddisk checks: [PASSED] 2015-03-12 03:59:24 Checking flash cache status..... 2015-03-12 03:59:27 Flashcache status normal: [PASSED] 2015-03-12 03:59:27 Checking that all FlashDisks are present... 2015-03-12 03:59:30 FlashDisk validation: [PASSED] 2015-03-12 03:59:30 Checking current flash cache mode..... 2015-03-12 03:59:31 Flashcache not already in target mode: [PASSED] 2015-03-12 03:59:31 All pre-req checks completed: [PASSED] 2015-03-12 03:59:32 dbm01cel01: flashcache size: 5.82122802734375T dbm01cel02: flashcache size: 5.82122802734375T dbm01cel03: flashcache size: 5.82122802734375T dbm01cel04: flashcache size: 5.82122802734375T dbm01cel05: flashcache size: 5.82122802734375T dbm01cel06: flashcache size: 5.82122802734375T dbm01cel07: flashcache size: 5.82122802734375T There are 7 storage cells to process. Troubleshoot if there are issues. ii. Now ENABLE the WBFC mode using the following command in NON-ROLLING Mode. [root@dbm01db01 stage]# ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteBack -o non-rolling ./setWBFC.sh: Using log directory 'setWBFC_log' ./setWBFC.sh: Log File 'setWBFC_log/setWBFC_66192_2015-03-12-04:01:28.log' created successfully 2015-03-12 04:01:28 Starting ./setWBFC.sh on dbm01db01 Version: 1.0.0.1.6.20140716 Command line options used: -g /root/cell_group -d /root/dbs_group -i (interractive) -o non-rolling -m WriteBack -t 21600 -x 0 2015-03-12 04:01:28 Performing pre-req checks..... 2015-03-12 04:01:28 Creating baseline inventory for griddisks 2015-03-12 04:01:34 Creating baseline inventory for flashdisks 2015-03-12 04:01:35 Creating baseline inventory for flashsize 2015-03-12 04:01:36 dcli present and in PATH. [PASSED] 2015-03-12 04:01:36 Checking cell nodes are valid storage servers... 2015-03-12 04:01:36 All cells are valid Exadata storage cells. 2015-03-12 04:01:36 Checking Exadata Storage Software Versions... 2015-03-12 04:01:40 Software versions of the following cells: dbm01cel01: 12.1.2.1.0.141206.1 [PASSED] dbm01cel02: 12.1.2.1.0.141206.1 [PASSED] dbm01cel03: 12.1.2.1.0.141206.1 [PASSED] dbm01cel04: 12.1.2.1.0.141206.1 [PASSED] dbm01cel05: 12.1.2.1.0.141206.1 [PASSED] dbm01cel06: 12.1.2.1.0.141206.1 [PASSED] dbm01cel07: 12.1.2.1.0.141206.1 [PASSED] 2015-03-12 04:01:40 Checking Grid Infrastructure Software Version... 2015-03-12 04:01:42 Grid Infrastructure version: 11.2.0.4.15 [PASSED] 2015-03-12 04:01:42 Checking for active ASM operations.... 2015-03-12 04:01:42 Check for no active ASM operations: [PASSED] 2015-03-12 04:01:42 Checking griddisk status across all cells.... 2015-03-12 04:01:44 All griddisks across all cells have asmdeactivationoutcome = Yes All griddisks across all cells are ONLINE Griddisk checks: [PASSED] 2015-03-12 04:01:44 Checking flash cache status..... 2015-03-12 04:01:45 Flashcache status normal: [PASSED] 2015-03-12 04:01:45 Checking that all FlashDisks are present... 2015-03-12 04:01:46 FlashDisk validation: [PASSED] 2015-03-12 04:01:46 Checking current flash cache mode..... 2015-03-12 04:01:47 Flashcache not already in target mode: [PASSED] 2015-03-12 04:01:47 All pre-req checks completed: [PASSED] 2015-03-12 04:01:48 dbm01cel01: flashcache size: 5.82122802734375T dbm01cel02: flashcache size: 5.82122802734375T dbm01cel03: flashcache size: 5.82122802734375T dbm01cel04: flashcache size: 5.82122802734375T dbm01cel05: flashcache size: 5.82122802734375T dbm01cel06: flashcache size: 5.82122802734375T dbm01cel07: flashcache size: 5.82122802734375T There are 7 storage cells to process. All pre-req checks are complete. Ready to proceed with changing the flash cache mode to WriteBack. OK to continue? [Y|y|N|n] :y Continuing.... 2015-03-12 04:02:07 Changing flash cache to WriteBack NON-ROLLING.... 2015-03-12 04:02:07 STEP 1: Dropping flashcache on all cells 2015-03-12 04:02:48 STEP 1: Completed sucessfully on all cells 2015-03-12 04:02:51 Skipping STEP 2: Inactivation grid disks not required. 2015-03-12 04:02:51 Skipping STEP 3: Shutdown of cellsrv not required. 2015-03-12 04:02:51 STEP 4: Set the flashCachMode to WriteBack on all cells 2015-03-12 04:02:51 STEP 4: Setting flashCacheMode to WriteBack 2015-03-12 04:02:51 STEP 4: Completed successfully on all cells. 2015-03-12 04:02:54 Skipping STEP 5: Restart of cellsrv not required. 2015-03-12 04:02:54 Skipping STEP 6: Activation of grid disk not required. 2015-03-12 04:02:54 STEP 7: Creating flashcache on all cells. 2015-03-12 04:03:50 STEP 7: Completed sucessfully on all cells. 2015-03-12 04:03:53 STEP 8: Verifying flashCacheMode on all cells. 2015-03-12 04:03:54 STEP 8: Completed sucessfully on all cells. Flash Cache mode is now WriteBack 2015-03-12 04:03:54 Validating inventory for griddisks 2015-03-12 04:03:56 Validation of griddisk: [PASSED] 2015-03-12 04:03:56 Validating inventory for flashdisks 2015-03-12 04:04:00 Validation of flashdisk: [PASSED] 2015-03-12 04:04:00 Validating inventory for flashsize 2015-03-12 04:04:02 Validation of flashsize: [PASSED] 2015-03-12 04:04:02 Setting flash cache to WriteBack completed successfully. [root@dbm01db01 stage]# DISABLE WBFC IN NON-ROLLING FASHION i. Run the following command to perform a prerequisite check. [root@dbm01db01 stage]# ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteThrough -o non-rolling -p ./setWBFC.sh: Using log directory 'setWBFC_log' ./setWBFC.sh: Log File 'setWBFC_log/setWBFC_10150_2015-05-11-09:05:33.log' created successfully 2015-05-11 09:05:33 Starting ./setWBFC.sh on dbm01db01 Version: 1.0.0.1.6.20140716 Command line options used: -g /root/cell_group -d /root/dbs_group -i (interractive) -o non-rolling -m WriteThrough -p (Perform pre-req checks only) -t 21600 -x 0 2015-05-11 09:05:33 Performing pre-req checks..... 2015-05-11 09:05:33 Creating baseline inventory for griddisks 2015-05-11 09:05:38 Creating baseline inventory for flashdisks 2015-05-11 09:05:42 Creating baseline inventory for flashsize 2015-05-11 09:05:46 dcli present and in PATH. [PASSED] 2015-05-11 09:05:46 Checking cell nodes are valid storage servers... 2015-05-11 09:05:47 All cells are valid Exadata storage cells. 2015-05-11 09:05:47 Checking Exadata Storage Software Versions... 2015-05-11 09:05:50 Software versions of the following cells: dbm01cel01: 12.1.2.1.1.150316.2 [PASSED] dbm01cel02: 12.1.2.1.1.150316.2 [PASSED] dbm01cel03: 12.1.2.1.1.150316.2 [PASSED] dbm01cel04: 12.1.2.1.1.150316.2 [PASSED] dbm01cel05: 12.1.2.1.1.150316.2 [PASSED] dbm01cel06: 12.1.2.1.1.150316.2 [PASSED] dbm01cel07: 12.1.2.1.1.150316.2 [PASSED] 2015-05-11 09:05:50 Checking Grid Infrastructure Software Version... 2015-05-11 09:05:52 Grid Infrastructure version: 11.2.0.4.15 [PASSED] 2015-05-11 09:05:52 Checking for active ASM operations.... 2015-05-11 09:05:53 Check for no active ASM operations: [PASSED] 2015-05-11 09:05:53 Checking griddisk status across all cells.... 2015-05-11 09:05:55 All griddisks across all cells have asmdeactivationoutcome = Yes All griddisks across all cells are ONLINE Griddisk checks: [PASSED] 2015-05-11 09:05:55 Checking flash cache status..... 2015-05-11 09:05:56 Flashcache status normal: [PASSED] 2015-05-11 09:05:56 Checking that all FlashDisks are present... 2015-05-11 09:05:58 FlashDisk validation: [PASSED] 2015-05-11 09:05:58 Checking current flash cache mode..... 2015-05-11 09:05:59 Flashcache not already in target mode: [PASSED] 2015-05-11 09:05:59 All pre-req checks completed: [PASSED] 2015-05-11 09:06:00 dbm01cel01: flashcache size: 5.82122802734375T dbm01cel02: flashcache size: 5.82122802734375T dbm01cel03: flashcache size: 5.82122802734375T dbm01cel04: flashcache size: 5.82122802734375T dbm01cel05: flashcache size: 5.82122802734375T dbm01cel06: flashcache size: 5.82122802734375T dbm01cel07: flashcache size: 5.82122802734375T There are 7 storage cells to process. Troubleshoot if any issues. ii. Now DISABLE the WBFC mode using the following command in NON-ROLLING Mode. This will command will change the Write-Back mode to Write-Through mode. [root@dbm01db01 stage]# ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteThrough -o non-rolling ./setWBFC.sh: Using log directory 'setWBFC_log' ./setWBFC.sh: Log File 'setWBFC_log/setWBFC_20103_2015-05-11-09:06:32.log' created successfully 2015-05-11 09:06:32 Starting ./setWBFC.sh on dbm01db01 Version: 1.0.0.1.6.20140716 Command line options used: -g /root/cell_group -d /root/dbs_group -i (interractive) -o non-rolling -m WriteThrough -t 21600 -x 0 2015-05-11 09:06:32 Performing pre-req checks..... 2015-05-11 09:06:32 Creating baseline inventory for griddisks 2015-05-11 09:06:33 Creating baseline inventory for flashdisks 2015-05-11 09:06:34 Creating baseline inventory for flashsize 2015-05-11 09:06:35 dcli present and in PATH. [PASSED] 2015-05-11 09:06:35 Checking cell nodes are valid storage servers... 2015-05-11 09:06:35 All cells are valid Exadata storage cells. 2015-05-11 09:06:35 Checking Exadata Storage Software Versions... 2015-05-11 09:06:39 Software versions of the following cells: dbm01cel01: 12.1.2.1.1.150316.2 [PASSED] dbm01cel02: 12.1.2.1.1.150316.2 [PASSED] dbm01cel03: 12.1.2.1.1.150316.2 [PASSED] dbm01cel04: 12.1.2.1.1.150316.2 [PASSED] dbm01cel05: 12.1.2.1.1.150316.2 [PASSED] dbm01cel06: 12.1.2.1.1.150316.2 [PASSED] dbm01cel07: 12.1.2.1.1.150316.2 [PASSED] 2015-05-11 09:06:39 Checking Grid Infrastructure Software Version... 2015-05-11 09:06:40 Grid Infrastructure version: 11.2.0.4.15 [PASSED] 2015-05-11 09:06:40 Checking for active ASM operations.... 2015-05-11 09:06:40 Check for no active ASM operations: [PASSED] 2015-05-11 09:06:40 Checking griddisk status across all cells.... 2015-05-11 09:06:44 All griddisks across all cells have asmdeactivationoutcome = Yes All griddisks across all cells are ONLINE Griddisk checks: [PASSED] 2015-05-11 09:06:44 Checking flash cache status..... 2015-05-11 09:06:45 Flashcache status normal: [PASSED] 2015-05-11 09:06:45 Checking that all FlashDisks are present... 2015-05-11 09:06:46 FlashDisk validation: [PASSED] 2015-05-11 09:06:46 Checking current flash cache mode..... 2015-05-11 09:06:47 Flashcache not already in target mode: [PASSED] 2015-05-11 09:06:47 All pre-req checks completed: [PASSED] 2015-05-11 09:06:48 dbm01cel01: flashcache size: 5.82122802734375T dbm01cel02: flashcache size: 5.82122802734375T dbm01cel03: flashcache size: 5.82122802734375T dbm01cel04: flashcache size: 5.82122802734375T dbm01cel05: flashcache size: 5.82122802734375T dbm01cel06: flashcache size: 5.82122802734375T dbm01cel07: flashcache size: 5.82122802734375T There are 7 storage cells to process. All pre-req checks are complete. Ready to proceed with changing the flash cache mode to WriteThrough. OK to continue? [Y|y|N|n] :y Continuing.... 2015-05-11 09:06:53 Changing flash cache to WriteThrough NON-ROLLING.... 2015-05-11 09:06:53 FLASH CACHE FLUSH MONITOR 2015-05-11 09:06:56 Flush flash disk working: 0 Flush flash disk completed: 0 Flush flash disk with errors: 0 Total flash disk: 28 2015-05-11 09:06:56 2015-05-11 09:06:54 dbm01cel01: FC_BY_DIRTY 760 MB dbm01cel02: FC_BY_DIRTY 760 MB dbm01cel03: FC_BY_DIRTY 737 MB dbm01cel04: FC_BY_DIRTY 731 MB dbm01cel05: FC_BY_DIRTY 737 MB dbm01cel06: FC_BY_DIRTY 774 MB dbm01cel07: FC_BY_DIRTY 725 MB 2015-05-11 09:07:12 Flush flash disk working: 8 Flush flash disk completed: 20 Flush flash disk with errors: 0 Total flash disk: 28 2015-05-11 09:07:12 FLASHCACHE FLUSH Error: Flush process prematurely aborted. FLASHCACHE FLUSH is not complete. Pleae cancel the flush with: alter flashcache all cancel flush 2015-05-11 09:07:12 STEP 1: Dropping flashcache on all cells 2015-05-11 09:07:56 STEP 1: Completed sucessfully on all cells 2015-05-11 09:07:59 Skipping STEP 2: Inactivation of grid disks not required. 2015-05-11 09:07:59 Skipping STEP 3: Shutdown of cellsrv not requried. 2015-05-11 09:07:59 STEP 4: Set the flashCachMode to WriteThrough on all cells 2015-05-11 09:07:59 STEP 4: Setting flashCacheMode to WriteThrough 2015-05-11 09:08:00 STEP 4: Completed successfully on all cells. 2015-05-11 09:08:03 Skipping STEP 5: Restart of cellsrv not required. 2015-05-11 09:08:03 Skipping STEP 6: Activation of grid disk not required. 2015-05-11 09:08:03 STEP 7: Creating flashcache on all cells. 2015-05-11 09:08:19 STEP 7: Completed sucessfully on all cells. 2015-05-11 09:08:22 STEP 8: Verifying flashCacheMode on all cells. 2015-05-11 09:08:23 STEP 8: Completed sucessfully on all cells. Flash Cache mode is now WriteThrough 2015-05-11 09:08:23 Validating inventory for griddisks 2015-05-11 09:08:24 Validation of griddisk: [PASSED] 2015-05-11 09:08:24 Validating inventory for flashdisks 2015-05-11 09:08:25 Validation of flashdisk: [PASSED] 2015-05-11 09:08:25 Validating inventory for flashsize 2015-05-11 09:08:26 Validation of flashsize: [PASSED] 2015-05-11 09:08:26 Setting flash cache to WriteThrough completed successfully Verify Flash Cache Mode Execute the following dcli command to verify the Flash Cache mode. [root@dbm01db01 stage]# dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode" dbm01cel01: WriteBack dbm01cel02: WriteBack dbm01cel03: WriteBack dbm01cel04: WriteBack dbm01cel05: WriteBack dbm01cel06: WriteBack dbm01cel07: WriteBack Measure WBFC Statistics Flash cache related metrics in Oracle Database 11.2.0.4: · Cell Flash Cache Read Hits à indicates blocks read from flash cache · Cell Overwrites in Flash Cache à Indicates block already exist in flash cache · Cell Partial Writes in Flash Cache à indicates only partial blocks cached in flash · Cell Writes to Flash Cache à indicates blocks written to flash cache Both Flash Cache Read and Write hits were accumulated in one metrics “Read Hits” in Oracle Database 11.2.0.3 and 12.1.0.1. But in 11.2.0.4 2we have separate metrics for both Read and Write metrics. Query V$sesstat or V$sysstat (gv$sesstat or gv$sysstat) to measure Read and Write hits to flash cache. SQL> set lines 200 SQL> set pages 200 SQL> break on inst_id SQL> column name for a60 SQL> select inst_id,name, value from gv$sysstat where name like lower('%&1%') order by 1,2; Enter value for 1: cell%flash old 1: select inst_id,name, value sys_value from gv$sysstat where lower(name) like lower('%&1%') order by 1,2 new 1: select inst_id,name, value sys_value from gv$sysstat where lower(name) like lower('%cell%flash%') order by 1,2 INST_ID NAME VALUE ---------- ------------------------------------------------------------ ---------- 1 cell flash cache read hits 79731401 cell overwrites in flash cache 34387629 cell partial writes in flash cache 27587 cell writes to flash cache 67534230 2 cell flash cache read hits 71593887 cell overwrites in flash cache 31209935 cell partial writes in flash cache 91402 cell writes to flash cache 50465666 3 cell flash cache read hits 95650309 cell overwrites in flash cache 42161201 cell partial writes in flash cache 89464 cell writes to flash cache 72504903 4 cell flash cache read hits 116445502 cell overwrites in flash cache 29630935 cell partial writes in flash cache 44560 cell writes to flash cache 45760963 Important Note The HDD controller on each Exadata Storage cell periodically performs battery learn cycle of the controller battery. During this operation, Write-Back changes to Write-Through mode. Write-Through is slower than Write-Back mode. Once the “battery learn cycle” is completed, it will automatically changes to Write-Back mode. If it doesn't change to Write-Back, there may be a problem with the controller battery. Work with Oracle Support to replace the batteries. For Oracle Exadata Storage Server Software version (ESSS) 11.2.1.3.0 and above this operation occurs every quarter. For ESSS version earlier than 11.2.1.3.0, this operation occurs every month. An informational alert similar to below will be generated by the Exadata Storage Servers "All hard disk drives may temporarily enter WriteThrough caching mode as part of the learn cycle. Disk write throughput might be temporarily lower during this time. The flash drives are not affected. The battery learn cycle is a normal maintenance activity that occurs quarterly and runs for approximately 1 to 12 hours. Note that many learn cycles do not require entering WriteThrough caching mode. When the disk controller cache returns to the normal WriteBack caching mode, an additional informational alert will be sent." BUGs 1. Exadata WriteBack Flashcache requires 11.2.3.2.1 image (Doc ID 1557421.1) 2. Systems with WriteBack Smart Flash Cache (WBFC) enabled running into unnecessary block repair during resilvering could cause data loss (Doc ID 1594155.1) 3. Bug 14506367 - Overcounted 'physical read optimized' statistic with writeback flash (Doc ID 14506367.8) Useful Commands Pre-requisites check example · Pre-requisites checks for enabling write back flash cache in a rolling fashion: # ./setWBFC.sh -g ~/cell_group -l setWBFC_log -m WriteBack -o rolling -p · Pre-requisites checks for enabling write back flash cache in a non-rolling fashion: # ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -l setWBFC_log -m WriteBack -o non-rolling -p · Pre-requisites checks for disabling write back flash cache in a rolling fashion: # ./setWBFC.sh -g ~/cell_group -l setWBFC_log -m WriteThrough -o rolling -p · Pre-requisites checks for disabling write back flash cache in a non-rolling fashion: # ./setWBFC.sh -g ~/cell_group –d ~/dbs_group -l setWBFC_log -m WriteThrough -o non-rolling –p Enable WBFC example · Enable WBFC in rolling fashion: # ./setWBFC.sh -g ~/cell_group -i -l setWBFC_log -m WriteBack -o rolling · Enable WBFC in non-rolling fashion: # ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteBack -o non-rolling -p Disable WBFC example · Disable WBFC in rolling fashion: # ./setWBFC.sh -g ~/cell_group -i -l setWBFC_log -m WriteThrough -o rolling · Disable WBFC in non-rolling fashion: # ./setWBFC.sh -d ~/dbs_group -g ~/cell_group -i -l setWBFC_log -m WriteThrough -o non-rolling –p Conclusion In this article we have learned how to ENABLE and DISABLE WBFC using Automation script in different ways. References 1. Exadata Write-Back Flash Cache - FAQ (Doc ID 1500257.1) 2. http://blog.tanelpoder.com/2013/12/04/cell-flash-cache-read-hits-vs-cell-writes-to-flash-cache-statistics-on-exadata/

Viewing all articles
Browse latest Browse all 1814

Latest Images

Trending Articles



Latest Images

<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>