Introduction After migrating database(s) to Exadata Database Machine you must configure database backups. Exadata uses Automatic Storage Management (ASM) for storing database files. Using RMAN is the only option to perform database files backup in Exadata. You can store RMAN backups in Fast Recovery Area (FRA) or External backup media (Example: ZFS Storage Appliance) based on your business requirement. If there is sufficient space in +RECO ASM disk group you can configure RMAN backup to FRA. If the customer has purchased ZFS Storage Appliance you can configure RMAN backups to ZFS file system. Oracle Sun ZFS Storage Appliance Oracle ZFS backup Appliance is a high performance backup solution for Oracle engineered systems (Exadata, SuperCluster and ODA) that reduces backup and recovery time objectives. The Oracle ZFS Backup Appliance is fully tested, validated and supported backup appliance specifically tuned for Oracle engineered systems backup and recovery. ZFS Storage Appliance benefits are: Greater Performance Cost effectiveness Simplified management Reduced risk The Oracle ZFS Storage Appliance is ideal for backing up the databases running on Exadata Database machine, Oracle SuperCluster and Oracle Database Appliance (ODA). Many Customers purchased Exadata uses ZFS for backing up databases. For more details on Oracle Sun ZFS Storage Appliance, please refer to following links: http://www.oracle.com/us/products/servers-storage/storage/nas/zfs-backup-appliance/zfs-backup-appliance-ds-1579018.pdf http://www.oracle.com/technetwork/database/features/availability/maa-wp-dbm-zfs-backup-1593252.pdf ESBU Utility The Oracle Engineered Systems Backup Utility (ESBU) formerly known as "Exadata Backup Configuration Utility" for Oracle ZFS Storage Appliance configures the Oracle ZFS Storage Appliance and the database compute nodes of a particular engineered system (Exadata, SuperCluster or ODA) through a command-line wizard. Exadata backup Configuration Utility was renamed to "Engineered Systems Backup Utility" starting with ESBU version 2.0.0 ESBU utility automates the manual steps that were required to configure ZFS Storage Appliance. This enforces the best practices and configuration. You Download ESBU utility from OTN at: http://www.oracle.com/technetwork/server-storage/sun-unified-storage/downloads/zfssa-plugins-1489830.html ESBU utility performs the entire Configuration process in three steps. Oracle ZFS Storage Appliance configuration Performed by Storage Admin or Exadata DBA In this step a project including NFS shares is configured in a user-selected storage pool. Engineered System nodes configuration Performed by System Admin or Exadata DBA In this step the client NFS and Oracle Direct NFS (dNFS) are configured. RMAN run block script generation Performed by DBA or Exadata DBA In this step RMAN scripts are generated in this step. Refer to the ESBU user guide for more details. In this article I will demonstrate how to Configure ZFS shares on Exadata Database machines using ESBU utility after the ZFS Storage Appliance is Installed and Configured by Storage Administrator or Oracle ACS. At end of the ESBU execution, the ZFS shares will be mounted on all the Exadata Compute nodes. ESBU Download and Installation Instructions Download the latest ESBU utility zip file from the OTN. Copy the ESBU zip file to the directory on the Engineered system. Here I am copying zip file to Exadata Compute node 1. [root@dbm01db01 stage] mkdir -p /u01/stage [root@dbm01db01 stage] cd /u01/stage [root@dbm01db01 stage] pwd /u01/stage Unzip the esbu zip file. The esbu directory is created and contains the utility scripts. [root@cay02db01 stage]# unzip esbu_211_2a.zip Archive: esbu_211_2a.zip creating: esbu/ inflating: esbu/create_project_shares.sh inflating: esbu/destroy_project_shares.sh inflating: esbu/eng_system_config.py inflating: esbu/eng_system_storage_exp_config.py inflating: esbu/esbu inflating: esbu/esbu_common.py inflating: esbu/esbu_config.py inflating: esbu/esbu_startup.py inflating: esbu/esbu_storage_exp inflating: esbu/exadata_bcu inflating: esbu/exadata_config.py inflating: esbu/get_system_info.sh inflating: esbu/install_key inflating: esbu/list_interfaces.sh inflating: esbu/list_pools.sh creating: esbu/log/ inflating: esbu/oda_config.py inflating: esbu/oda_exp_config.py inflating: esbu/README inflating: esbu/rest_api_client.py creating: esbu/rman/ inflating: esbu/rman_run_block.py inflating: esbu/s7k_config.py inflating: esbu/s7k_config_esbu.py inflating: esbu/s7k_config_exp.py inflating: esbu/s7k_rest.py inflating: esbu/sc_config.py inflating: esbu/set_ipmp.sh creating: esbu/simplejson/ inflating: esbu/simplejson/__init__.py inflating: esbu/simplejson/decoder.py inflating: esbu/simplejson/encoder.py inflating: esbu/simplejson/LICENSE.txt inflating: esbu/simplejson/scanner.py creating: esbu/template/ inflating: esbu/uninstall_key inflating: esbu/verify_project_shares.sh [root@cay03db01 stage]# ls -ltr total 68 -rw-r--r-- 1 root root 62263 Mar 15 05:33 esbu_211_2a.zip drwxr-x--- 6 root root 4096 Mar 24 21:30 esbu Assumption A fully functional ZFS Storage Appliance. ZFS Storage Appliance Hostnames. Exadata Database Machine is installed and configured. Exadata Database Machine Compute node names. Root user password for ZFS Storage appliance. Default root password is changeme. Root user password for Exadata Compute nodes. Default root password is welcome1. ESBU utility is downloaded and unzipped. Environment Exadata Model X5 Half Rack 4TB Exadata Components Storage Cell (7), Compute node (4) & Infiniband Switch (2) Exadata Storage cells DB01CEL01 – DB01CEL07 Exadata Compute nodes DB01DB01 – DB01DB04 Exadata Software Version 12.1.2.3.0 Exadata DB Version 11.2.0.4 BP20 Exadata Infiniband Version 2.1.6-2 ZFS Appliance Name zsanas01 & zsanas02 (2 Disk Controllers and 2 Disk Shelves) ZFS Appliance Model Oracle ZFS Storage ZS3-2 Appliance Type Sun ZFS Storage 7330 Appliance Version 2013.06.05.4.2,1-1.1 ESBU Utility Execution Steps ZFS Storage Appliance and Exadata details ZFS Storage Appliance : zsanas01/zsanas02 Exadata : dm01 Compute Nodes : dm01db01, dm02db02, dm03db03, dm04db04 Update /etc/resolv.conf to include the ZFS Storage Appliance domain name. This will allow you to SSH to ZFS Storage Appliance with or without domain name. Update the /resolv.conf file on all Compute Nodes [root@dm01db01 ]# vi /etc/resolv.conf Verify you can SSH using full qualified hostname [root@dm01db01 esbu]# ssh zsanas01.mydomain.com Password: [root@dm01db01 esbu]# ssh zsanas02.mydomain.com Password: Install SSH keys On Exadata compute node 1, install the SSH key on ZFS Storage Appliance and verify the passwordless authentication by logging in to the Storage Appliance using SSH. Navigate to the directory where ESBU utility is installed. [root@dm01db01 ]# cd /u01/stage/esbu/ [root@dm01db01 esbu]# ./install_key root@zsanas01.mydomain.com root@zsanas01.mydomain.com Using the existing /root/.ssh/id_rsa.pub Password: type = RSA (uncommitted) comment = eng_sys_backup_utility (uncommitted) key = LXTQd1LLdWxP+VY7BbxKxIBSqiMVwjeJzN+X2SaIwGaEYQVY1XqwCgDHi6N2g74GSA842oDClQDB11VT2J1Zi8O+yeLZd/hOz3Y1XsuygRLXnZDjSXM8VHe1YVQ2HmcUfs+gcVccRtBfmXdYZMO7rZVq0I+DmMquf/m2FZKSaTRq8X8FHS+eC39OZ2Ey4SDGZpy6or/zUEX1nON//gJ6CtSUJDBeQ7Pk0BINXT1W22gK3Wf/fwTUqNot5g9Q== (uncommitted) Verifying connection to the second head: zsanas02 The authenticity of host 'zsanas02 (167.68.10.1)' can't be established. RSA key fingerprint is ce:d4:3f:e2:de:54:bf:a1:dc:bb:34:ec:56:9b:b1:18. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'zsanas02' (RSA) to the list of known hosts. Key installation complete Verify passwordless authentication to ZFS Storage Appliance. [root@dm01db01 esbu]# ssh zsanas01.mydomain.com date 2015-3-24 20:34:18 [root@dm01db01 esbu]# ssh zsanas02.mydomain.com date 2015-3-24 20:34:23 [root@dm01db01 ~]# ssh zsanas01 date 2015-4-16 13:44:06 [root@dm01db01 ~]# ssh zsanas02 date 2015-4-16 13:44:10 On Exadata Compute nodes, ensure that “rcpbind” and nfslock” Linux services are enabled and running. [root@dm01db01 ~]# dcli -g ~/dbs_group -l root 'service rpcbind start' dm01db01: Starting rpcbind: [ OK ] dm01db02: Starting rpcbind: [ OK ] dm01db03: Starting rpcbind: [ OK ] dm01db04: Starting rpcbind: [ OK ] [root@dm01db01 ~]# dcli -g ~/dbs_group -l root 'service nfslock start' dm01db01: Starting NFS statd: [ OK ] dm01db02: Starting NFS statd: [ OK ] dm01db03: Starting NFS statd: [ OK ] dm01db04: Starting NFS statd: [ OK ] [root@dm01db01 ~]# dcli -g ~/dbs_group -l root 'chkconfig rpcbind on' [root@dm01db01 ~]# dcli -g ~/dbs_group -l root 'chkconfig --list | grep rpcbind' dm01db01: rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off dm01db02: rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off dm01db03: rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off dm01db04: rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@dm01db01 ~]# dcli -g ~/dbs_group -l root 'chkconfig nfslock on' [root@dm01db01 ~]# dcli -g ~/dbs_group -l root 'chkconfig --list | grep nfslock' dm01db01: nfslock 0:off 1:off 2:on 3:on 4:on 5:on 6:off dm01db02: nfslock 0:off 1:off 2:on 3:on 4:on 5:on 6:off dm01db03: nfslock 0:off 1:off 2:on 3:on 4:on 5:on 6:off dm01db04: nfslock 0:off 1:off 2:on 3:on 4:on 5:on 6:off Execute the ESBU utility from the installation directory. ESBU utility runs in the interactive mode and asks the details about ZFS Storage Appliance and Exadata Database Machine. NOTE: The lines highlighted in GREEN are the ones where you supply the input to the ESBU utility [root@dm01db01 esbu]# ./esbu This utility configures Engineered System database backup on Oracle ZFS Storage Appliance The following types of Engineered Systems are supported for database backup: 0. Exadata 1. SuperCluster 2. Oracle Database Appliance (ODA) Enter the number corresponding to the type of Engineered System database to be backed up: 0 -> Here I selected 0 for Exadata Step 1 Configure Oracle ZFS Storage Appliance In this step a project and shares will be created in the appliance. Prior to this step, a storage pool and a network interface must be configured first using the BUI of the appliance (https://appliance-mgt-ip:215). To complete this step successfully, the following information will be required: - Host name and user name of the appliance - Project name to be created on Oracle ZFS Storage Appliance. - IP addresses of the Engineered System to have NFS read/write access Continue? (y|n) y -> enter y and hit return Enter the host name of the appliance: zsanas01 -> Enter the hostname of the first ZFS Storage Appliance and hit return Enter the user name of the appliance: root -> enter user name as root and hit return Enter the project name to be created on Oracle ZFS Storage Appliance: dm01 -> enter the 4 character of Exadata name and hit return Enter the IPs of the Engineered System for NFS rw access: List all IPs or use netmasks. Separate multiple entries by comma (,) Example: 10.80.36.0/24,10.80.37.81/0 (netmask is required) 192.168.10.0/24 -> enter the private IP of the Exadata Compute node and hit return Obtaining system info from root@zsanas01 ... Type: Sun ZFS Storage 7420 Version: 2013.06.05.2.13,1-1.1 Cluster: true Cluster State: AKCS_CLUSTERED Continue? (y|n) y -> Verify the information printed above. Enter y and hit return Obtaining available pools from root@zsanas01 ... Available storage pools on zsanas01: POOL PROFILE STATUS 1 - Pool-0 raidz2_nspf online 0 - Desired pool does not exist. Quit Enter digit to select a pool: 1 -> enter 1 to select Pool-0 and hit return Pool-0 will be used for the database backup Obtaining available network interfaces from zsanas01 ... Available network interfaces to access Pool-0: INTERFACE ADDRESS STATE LABEL igb0 166.68.10.1/24 up E01_igb0_MGMT ipmp1 192.168.10.200/24 up e01_ibp0_ibp3_exa_ipmp0 ipmp1 192.168.10.202/24 up e01_ibp0_ibp3_exa_ipmp0 ixgbe0 10.214.35.38/29 up E01_ixgbe0_10GbE Enter the IP addesses to access the selected pool: List all IPs without netmasks. Separate multiple entries by comma (,) Example: 10.80.36.81,10.80.37.82 192.168.10.200,192.168.10.202 -> enter a set of Private IPs to access pool from first head. Here I am entering 2 Private IPs for high availability. The ZFS is configured with active/active IPMP 192.168.10.200,192.168.10.202 will be used to access Pool-0 Select synchronous write bias setting for shares: 0 - Throughput 1 - Latency Note: Throughput is recommended for backupset or image copy strategy. Latency is recommended for incremental merge backup strategy. Enter digit to select: 0 -> enter 0 to select Throughput and hit return Select another pool from the cluster peer head? (y|n) y -> enter y and hit return Obtaining system info from root@zsanas02 ... Type: Sun ZFS Storage 7420 Version: 2013.06.05.2.13,1-1.1 Cluster: true Cluster State: AKCS_CLUSTERED Continue? (y|n) y -> enter y and hit return Obtaining available pools from root@zsanas02 ... Available storage pools on zsanas02: POOL PROFILE STATUS 1 - Pool-1 raidz2_nspf online 0 - Desired pool does not exist. Quit Enter digit to select a pool: 1 -> enter 1 to select Pool-1 and hit return Pool-1 will be used for the database backup Obtaining available network interfaces from zsanas02 ... Available network interfaces to access Pool-1: INTERFACE ADDRESS STATE LABEL igb1 166.68.10.2/24 up E02_igb1_MGMT ipmp2 192.168.10.201/24 degraded e02_ibp1_ibp2_exa_ipmp0 ipmp2 192.168.10.203/24 degraded e02_ibp1_ibp2_exa_ipmp0 ixgbe1 10.214.35.46/29 up E02_ixgbe1_10GbE pffff_ibp0 0.0.0.0/8 offline e01_ibp0_int pffff_ibp1 0.0.0.0/8 up e02_ibp1_int pffff_ibp2 0.0.0.0/8 failed e02_ibp2_int pffff_ibp3 0.0.0.0/8 offline e01_ibp3_int Enter the IP addesses to access the selected pool: List all IPs without netmasks. Separate multiple entries by comma (,) Example: 10.80.36.81,10.80.37.82 192.168.10.201,192.168.10.203 -> enter a set of Private IPs to access pool from second head. Here I am entering 2 Private IPs for high availability. The ZFS is configured with active/active IPMP 192.168.10.201,192.168.10.203 will be used to access Pool-1 Verify configuration settings: Engineered System type: exadata ZFSSA project name: dm01 ZFSSA login: root ZFSSA shares per pool: 1 ZFSSA hostname - 1: zsanas01 ZFSSA pool - 1: Pool-0 ZFSSA interface - 1: 192.168.10.200,192.168.10.202 ZFSSA nfs exception: 192.168.10.0/24 ZFSSA hostname - 2: zsanas02 ZFSSA pool - 2: Pool-1 ZFSSA interface - 2: 192.168.10.201,192.168.10.203 The following NFS shares will be used to back up database: 192.168.10.200,192.168.10.202:/export/dm01/backup1 192.168.10.201,192.168.10.203:/export/dm01/backup2 Correct? (y|n) y -> enter y and hit return Verifying project and shares on Pool-0 ... Verifying project and shares on Pool-1 ... Verifying IPMP properties ... – zsanas01 Warning: IPMP Failure Detection Latency is set to 10000 milliseconds. Do you want to set the latency setting to the recommended value (5000 milliseconds)? (y|n) y -> enter y and hit return Warning: IPMP fail-back is enabled. Do you want to disable fail-back, which is the recommended setting? (y|n) y -> enter y and hit return Setting IPMP properties ... Creating project and shares on Pool-0 ... project created - dm01 share created - backup1 Verifying IPMP properties ... – zsanas02 Creating project and shares on Pool-1 ... project created - dm01 share created - backup2 Step 1 Configure Oracle ZFS Storage Appliance Completed Successfully. Continue to Step 2 Configure Engineered System Database Nodes? (y|n) y -> enter y and hit return Step 2 Configure Exadata Database Nodes In this step the Exadata database nodes will be configured. Prior to this step, the Oracle ZFS Storage Appliance must be configured by following Step 1. To complete this step successfully, the following information will be required: - Name of the database (DB_UNIQUE_NAME) to be backed up - Host name (node name) of an Exadata database node - Owner user and group name of the database and RMAN. The owner will own the NFS directories from the Oracle ZFS Storage Appliance where RMAN stores database backup. - File permission of the NFS directories where RMAN will store backup. - (Optional) Host IP addresses of the Exadata database nodes that will be used by Oracle Direct NFS (dNFS). The following configuration will be performed in this step: - Start database - DB_UNIQUE_NAME - Create mount points /zfssa/DB_UNIQUE_NAME/backup[1-8] - Add entries to /etc/fstab - Create and start /etc/init.d/zfssa_DB_UNIQUE_NAME to mount the NFS shares of the ZFSSA - Add entries to ORACLE_HOME/dbs/oranfstab - Enable Oracle Direct NFS - Add backup services - Restart database DB_UNIQUE_NAME Continue? (y|n) y -> enter y and hit return Enter the database name (DB_UNIQUE_NAME) to be backed up: dbm01 -> enter the Database unique name Enter the host name of an Exadata database node: dm01db01 -> enter the hostname of first compute node in the cluster Enter the owner user name of the database and RMAN. The user will own the NFS directories where RMAN stores backup: Hit Enter key to use the default (oracle) -> hit return to accept default Enter the owner group name of the database and RMAN. The group will own the NFS directories where RMAN stores backup: Hit Enter key to use the default (oinstall) -> hit return to accept default Enter the file mode of the NFS directories where RMAN stores backup: Hit Enter key to use the default (750) -> hit return to accept default Verify configuration settings: Database name: dbm01 Exadata hostname: dm01db01 DB Backup - owner: oracle:oinstall DB Backup - file mode: 750 Correct? (y|n) y -> enter y and hit return About to start configuring Exadata database nodes Continue? (y|n) y -> enter y and hit return Enter the ORACLE_HOME path: /u01/app/oracle/product/11.2.0.4/dbhome ORACLE_HOME is set to /u01/app/oracle/product/11.2.0.4/dbhome Obtaining database status - dbm01 Database is already running - dbm01 Instance dbm011 is running on node dm01db01 Instance dbm012 is running on node dm01db02 Instance dbm013 is running on node dm01db03 Instance dbm014 is running on node dm01db04 Continue? (y|n) y à enter y and hit return Performing system precheck - dm01db01 Performing connection precheck - dm01db01 Performing system precheck - dm01db02 Performing connection precheck - dm01db02 Performing system precheck - dm01db03 Performing connection precheck - dm01db03 Performing system precheck - dm01db04 Performing connection precheck - dm01db04 Continue? (y|n) y à enter y and hit return /bin/chown root:rpc /etc/hosts.allow /etc/hosts.deny - dm01db01 /bin/chmod 640 /etc/hosts.allow /etc/hosts.deny - dm01db01 Creating mount points - dm01db01 /zfssa/dbm01/backup1 - created /zfssa/dbm01/backup2 - created Adding mount points to fstab - dm01db01 Older version /etc/fstab is saved to /etc/fstab_20150416T085839 /etc/fstab: /zfssa/dbm01/backup1 - added /etc/fstab: /zfssa/dbm01/backup2 - added Creating /etc/init.d/zfssa_dbm01 - dm01db01 chkconfig zfssa_dbm01 on service zfssa_dbm01 start /etc/init.d/zfssa_dbm01 - created Does dm01db01 have more than one network interface available for the dNFS data paths? (y|n) y -> enter y and hit return – if more than one interface is configured on Exadata compute nodes Available network interfaces on dm01db01: 10.214.14.135 10.214.14.165 10.214.14.146 10.213.60.1 192.168.10.1 169.254.63.67 192.168.10.2 169.254.211.249 127.0.0.1 Press Enter to not select any local IP addresses Enter at most 4 local IP addresses of dm01db01 to be used by dNFS. Separate multiple entries by comma (,) Example: 10.80.36.150,10.80.36.151 192.168.10.1,192.168.10.2 -> enter a set of IPs if more than one interface is configured Updating oranfstab - dm01db01 /bin/chown oracle:oinstall /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db01 /bin/chmod 640 /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db01 /usr/bin/make -f /u01/app/oracle/product/11.2.0.4/dbhome/rdbms/lib/ins_rdbms.mk dnfs_on ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome - dm01db01 dNFS is enabled - dm01db01 /bin/chown root:rpc /etc/hosts.allow /etc/hosts.deny - dm01db02 /bin/chmod 640 /etc/hosts.allow /etc/hosts.deny - dm01db02 Creating mount points - dm01db02 /zfssa/dbm01/backup1 - created /zfssa/dbm01/backup2 - created Adding mount points to fstab - dm01db02 Older version /etc/fstab is saved to /etc/fstab_20150416T085925 /etc/fstab: /zfssa/dbm01/backup1 - added /etc/fstab: /zfssa/dbm01/backup2 - added Creating /etc/init.d/zfssa_dbm01 - dm01db02 chkconfig zfssa_dbm01 on service zfssa_dbm01 start /etc/init.d/zfssa_dbm01 - created Available network interfaces on dm01db02: 10.214.14.137 10.214.14.158 10.214.14.153 10.213.60.2 192.168.10.3 169.254.63.69 192.168.10.4 169.254.54.109 127.0.0.1 Press Enter to not select any local IP addresses Enter at most 4 local IP addresses of dm01db02 to be used by dNFS. Separate multiple entries by comma (,) Example: 10.80.36.150,10.80.36.151 192.168.10.3,192.168.10.4 -> enter a set of IPs if more than one interface is configured Updating oranfstab - dm01db02 /bin/chown oracle:oinstall /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db02 /bin/chmod 640 /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db02 /usr/bin/make -f /u01/app/oracle/product/11.2.0.4/dbhome/rdbms/lib/ins_rdbms.mk dnfs_on ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome - dm01db02 dNFS is enabled - dm01db02 /bin/chown root:rpc /etc/hosts.allow /etc/hosts.deny - dm01db03 /bin/chmod 640 /etc/hosts.allow /etc/hosts.deny - dm01db03 Creating mount points - dm01db03 /zfssa/dbm01/backup1 - created /zfssa/dbm01/backup2 - created Adding mount points to fstab - dm01db03 Older version /etc/fstab is saved to /etc/fstab_20150416T085943 /etc/fstab: /zfssa/dbm01/backup1 - added /etc/fstab: /zfssa/dbm01/backup2 - added Creating /etc/init.d/zfssa_dbm01 - dm01db03 chkconfig zfssa_dbm01 on service zfssa_dbm01 start /etc/init.d/zfssa_dbm01 - created Available network interfaces on dm01db03: 10.214.14.149 10.214.14.134 10.214.14.150 10.213.60.3 192.168.10.5 169.254.103.150 192.168.63.6 169.254.80.198 127.0.0.1 Press Enter to not select any local IP addresses Enter at most 4 local IP addresses of dm01db03 to be used by dNFS. Separate multiple entries by comma (,) Example: 10.80.36.150,10.80.36.151 192.168.10.5,192.168.10.6 -> enter a set of IPs if more than one interface is configured Updating oranfstab - dm01db03 /bin/chown oracle:oinstall /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db03 /bin/chmod 640 /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db03 /usr/bin/make -f /u01/app/oracle/product/11.2.0.4/dbhome/rdbms/lib/ins_rdbms.mk dnfs_on ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome - dm01db03 dNFS is enabled - dm01db03 /bin/chown root:rpc /etc/hosts.allow /etc/hosts.deny - dm01db04 /bin/chmod 640 /etc/hosts.allow /etc/hosts.deny - dm01db04 Creating mount points - dm01db04 /zfssa/dbm01/backup1 - created /zfssa/dbm01/backup2 - created Adding mount points to fstab - dm01db04 Older version /etc/fstab is saved to /etc/fstab_20150416T085957 /etc/fstab: /zfssa/dbm01/backup1 - added /etc/fstab: /zfssa/dbm01/backup2 - added Creating /etc/init.d/zfssa_dbm01 - dm01db04 chkconfig zfssa_dbm01 on service zfssa_dbm01 start /etc/init.d/zfssa_dbm01 - created Available network interfaces on dm01db04: 10.214.14.131 10.214.14.142 10.213.60.4 192.168.10.7 169.254.70.255 192.168.10.8 169.254.38.72 127.0.0.1 Press Enter to not select any local IP addresses Enter at most 4 local IP addresses of dm01db04 to be used by dNFS. Separate multiple entries by comma (,) Example: 10.80.36.150,10.80.36.151 192.168.10.7,192.168.10.8 -> enter a set of IPs if more than one interface is configured Updating oranfstab - dm01db04 /bin/chown oracle:oinstall /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db04 /bin/chmod 640 /u01/app/oracle/product/11.2.0.4/dbhome/dbs/oranfstab - dm01db04 /usr/bin/make -f /u01/app/oracle/product/11.2.0.4/dbhome/rdbms/lib/ins_rdbms.mk dnfs_on ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome - dm01db04 dNFS is enabled - dm01db04 /bin/chown oracle:oinstall /zfssa/dbm01/backup1 /bin/chmod 750 /zfssa/dbm01/backup1 /bin/chown oracle:oinstall /zfssa/dbm01/backup2 /bin/chmod 750 /zfssa/dbm01/backup2 Adding backup services ... ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl add service -d dbm01 -r dbm011 -a dbm012,dbm013,dbm014 -s dbm01_bkup1 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl start service -d dbm01 -s dbm01_bkup1 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl add service -d dbm01 -r dbm012 -a dbm011,dbm013,dbm014 -s dbm01_bkup2 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl start service -d dbm01 -s dbm01_bkup2 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl add service -d dbm01 -r dbm013 -a dbm011,dbm012,dbm014 -s dbm01_bkup3 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl start service -d dbm01 -s dbm01_bkup3 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl add service -d dbm01 -r dbm014 -a dbm011,dbm012,dbm013 -s dbm01_bkup4 ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome /u01/app/oracle/product/11.2.0.4/dbhome/bin/srvctl start service -d dbm01 -s dbm01_bkup4 About to restart the database dbm01 This step will interrupt the database. You may skip this step and restart the database manually later. Continue to restart the database? (y|n) y -> enter y if you OK to restart the databases Stopping database ... Database stopped successfully - dbm01 Starting database ... Database started successfully - dbm01 Stopping services ... Services stopped successfully - dbm01 Starting services ... Services started successfully - dbm01 Step 2 Configure Exadata Database Nodes Completed Successfully. Continue to Step 3 Generate RMAN Run Block Scripts? (y|n) y à enter y and hit return to move to next step Step 3 Generate RMAN Run Block Scripts In this step the RMAN run block scripts will be generated. Prior to this step the Oracle ZFS Storage Appliance and the Engineered System database nodes must be configured by following the Step 1 and 2 respectively. To complete this step successfully, the following information will be required: - Database login string (user/password@scan_address) or scan address Continue? (y|n) y -> enter y and hit return Enter the login string (user/password@scan_address) or scan address of dbm01: Example: sys/oracle@testdb-scan or testdb-scan dm01-scan Verify the login string for dbm01: Database login: dm01-scan Correct? (y|n) y -> enter y and hit return The following RMAN run block scripts are generated: /u01/app/oracle/software/esbu/rman/ dbm01_backupset_L0 - level 0 backupset dbm01_backupset_L1 - level 1 backupset dbm01_image_copy - level 0/1 image copy dbm01_inc_merge - incremental merge dbm01_restore_validate_backupset - restore validate for backupset dbm01_restore_validate_image_copy - restore validate for image copy The RMAN run block scripts are intended for samples only. Please review and modify the scripts as necessary for your requirements. Step 3 Generate RMAN Run Block Scripts Completed Successfully. This session is saved in /u01/stage/esbu/template/dm01 To continue next time, run esbu -t /u01/stage/esbu/template/dm01 This completes the ESBU utility execution. Now see if the mounts are visible on Exadata Compute nodes [root@dm01db01 ]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VGExaDb-LVDbSys1 30G 11G 18G 38% / tmpfs 252G 4.0K 252G 1% /dev/shm /dev/sda1 504M 43M 436M 9% /boot /dev/mapper/VGExaDb-LVDbOra1 97G 33G 64G 39% /u01 192.168.10.200:/export/dm01/backup1 82T 0 82T 0% /zfssa/dm01/backup1 192.168.10.201:/export/dm01/backup2 82T 1.0M 82T 1% /zfssa/dm01/backup2 Known issue Portmap is replaced by rpcbind on some linux distributions like OEL 6. Portmap is now started and stopped as part of the rpcbind service. You may not able to find the portmap service, but you can see rpcbind instead. [root@dm01db01 esbu]# cp -p eng_system_config.py eng_system_config.py_20160413 [root@dm01db01 esbu]# vi eng_system_config.py Note: comment lines 424-429 to remove portmap check The line numbers can be different in the latest ESBU utility. Make sure to comment the lines that call the portmap service. Conclusion In this article we have learnt about ESBU utility and how use it to mount the ZFS shares on Exadata Database Machines. ESBU utility replaces the manual steps that were required to configure ZFS Storage Appliance and enforces the best practices and configuration.
↧