One of the EBS RAC Node restarted and then database connections cannot be established on that specific node. The below error was reported when we were trying to connect database on the respective node. [applprod@ebsnode1 appl]$ sqlplus SQL*Plus: Release 10.1.0.5.0 - Production on Sun Feb 28 19:13:19 2016 Copyright (c) 1982, 2005, Oracle. All rights reserved. Enter user-name: apps@PROD Enter password: ERROR: ORA-12516: TNS:listener could not find available handler with matching protocol Issue: >> On Investigation we found there are two listener services running on that node: [oraprod@ebsnode1 PROD1_ebsnode1]$ ps -ef | grep lsn oraprod 4674 1 0 18:57 ? 00:00:00 /u01/grid/11.2.0.3/bin/tnslsnr LISTENER_SCAN1 -inherit oraprod 4701 1 0 18:57 ? 00:00:00 /u01/grid/11.2.0.3/bin/tnslsnr LISTENER -inherit oraprod 5956 5076 0 19:12 pts/2 00:00:00 grep lsn [oraprod@ebsnode1 PROD1_ebsnode1]$ Cause: >> The listener from RDBMS Home did not started after node reboot. Solution: >> Start Listener process from RDBMS Home and verify the connection. [oraprod@ebsnode1 PROD1_ebsnode1]$ lsnrctl start PROD LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 28-FEB-2016 19:14:32 Copyright (c) 1991, 2011, Oracle. All rights reserved. Starting /u01/oraprod/PROD/11.2.0/bin/tnslsnr: please wait... TNSLSNR for Linux: Version 11.2.0.3.0 - Production System parameter file is /u01/oraprod/PROD/11.2.0/network/admin/PROD1_ebsnode1/listener.ora Log messages written to /u01/oraprod/PROD/11.2.0/log/diag/tnslsnr/ebsnode1/prod/alert/log.xml Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.26)(PORT=1524))) Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.21)(PORT=1524))) Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ebsnode1-vip.oralabs.com)(PORT=1524)(IP=FIRST))) STATUS of the LISTENER ------------------------ Alias PROD Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production Start Date 28-FEB-2016 19:14:47 Uptime 0 days 0 hr. 0 min. 16 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/oraprod/PROD/11.2.0/network/admin/PROD1_ebsnode1/listener.ora Listener Log File /u01/oraprod/PROD/11.2.0/log/diag/tnslsnr/ebsnode1/prod/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.26)(PORT=1524))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.21)(PORT=1524))) Services Summary... Service "PROD1" has 1 instance(s). Instance "PROD1", status UNKNOWN, has 1 handler(s) for this service... The command completed successfully [oraprod@ebsnode1 PROD1_ebsnode1]$ >> Check listener processes: [oraprod@ebsnode1 PROD1_ebsnode1]$ ps -ef | grep lsn oraprod 4674 1 0 18:57 ? 00:00:00 /u01/grid/11.2.0.3/bin/tnslsnr LISTENER_SCAN1 -inherit oraprod 4701 1 0 18:57 ? 00:00:00 /u01/grid/11.2.0.3/bin/tnslsnr LISTENER -inherit oraprod 6025 1 0 19:14 ? 00:00:00 /u01/oraprod/PROD/11.2.0/bin/tnslsnr PROD -inherit oraprod 6105 5076 0 19:16 pts/2 00:00:00 grep lsn [oraprod@ebsnode1 PROD1_ebsnode1]$ >> Check DB Connection: Enter user-name: apps@PROD Enter password: Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> So we should ensure all services started properly after cluster nodes restarted or rebooted. Thanks for reading regards, X A H E E R
↧