While trying to query the management repository path using OCLUMON utility, I was receiving the following error. ---// ---// error while accessing management repository //--- ---// myracserver1 {/home/oracle}: $GRID_HOME/bin/oclumon manage -get reppath Connection Error. Could not get RepPath. myracserver1 {/home/oracle}: Trying to issue dumpnodeview command, generated following errors. ---// ---// error while running dumpnodeview //--- ---// myracserver1 {/home/oracle}: $GRID_HOME/bin/oclumon dumpnodeview -allnodes CRS-9118-Grid Infrastructure Management Repository connection error ORA-12514: TNS:listener does not currently know of service requested in connect descriptor From the error, it is evident that OCLUMON utility is not able to establish a connection with the management repository database. However, there is not enough information available in the error messages to draw any conclusion. We need to find more details related to this error. We know that, it is the Cluster Logger Service (ologgerd) which communicates with the management repository; which means we can find additional details about this error in the ologgerd (Cluseter Logger Service) trace file. To be able to find the ologgerd service trace file, we need to first identify the node in which ologgerd service is running (We can have one ologgerd service per 32 cluster nodes). We can do that using OCLUMON utility as shown below. ---// ---// locating node hosting the logger service //--- ---// myracserver1 {/home/oracle}: $GRID_HOME/bin/oclumon manage -get mylogger Logger = myracserver2 We have the ologgerd service running on node myracserver2 and we need to login to that node, to view the trace logs related to the connection error. Let’s login to the cluster node myracserver2. ---// ---// login to node hosting ologgerd service //--- ---// myracserver1 {/home/oracle}: ssh myracserver2 Last login: Tue Feb 23 03:06:33 2016 from myracserver1 myracserver2 {/home/oracle}: ---// ---// verify ologgerd service is running there //--- ---// myracserver2 {/home/oracle}: ps -ef | grep ologgerd root 2814 1 1 Jan26 ? 08:09:35 /app/grid/12.1.0.2/bin/ologgerd -M -d /app/grid/12.1.0.2/crf/db/myracserver2 oracle 32564 32235 0 09:37 pts/0 00:00:00 grep ologgerd As we know, with Oracle 12c all the Clusterware logs are centralized under the $ADR_BASE/diag/crs/`hostname`/crs location, we can find the ologgerd service trace files under this centralized log directory. Let’s find the ologgerd service trace file ---// ---// querying adrci to find the trace home //--- ---// myracserver2 {/home/oracle}: adrci ADRCI: Release 12.1.0.2.0 - Production on Fri Feb 26 09:39:39 2016 Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved. ADR base = "/app/oracle" adrci> show homes ADR Homes: diag/rdbms/_mgmtdb/-MGMTDB diag/tnslsnr/myracserver2/mgmtlsnr diag/crs/myracserver2/crs adrci> exit ---// ---// moving to clusterware trace directory //--- ---// myracserver2 {/home/oracle}: cd /app/oracle/diag/crs/myracserver2/crs/trace myracserver2 {/app/oracle/diag/crs/myracserver2/crs/trace}: ---// ---// locating ologgerd service trace file //--- ---// myracserver2 {/app/oracle/diag/crs/myracserver2/crs/trace}: ls -lrt ologgerd.trc -rw-rw---- 1 root oinstall 804345 Feb 26 09:36 ologgerd.trc We have now located the ologgerd service trace file, lets find out the error details from this trace file. ---// ---// viewing additional error details from ologgerd.trc //--- ---// 2016-02-26 09:47:36.795608 : CRFOCI:2608355072: crfoci_conn_create: Connection string discoverd 2016-02-26 09:47:36.820389 : CRFOCI:2608355072: crfoci_conn_create: Trying to connect to (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp)(HOST = 192.168.230.10)(PORT = 1521))(CONNECT_DATA = (SERVICE_NAME = my_rac_cluster))). 2016-02-26 09:51:10.616175 : CRFOCI:2608355072: crfoci_conn_create: Connection string discoverd 2016-02-26 09:51:10.659797 : CRFOCI:2608355072: crfoci_conn_create: Trying to connect to (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp)(HOST = 192.168.230.10)(PORT = 1521))(CONNECT_DATA = (SERVICE_NAME = my_rac_cluster))) . 2016-02-26 09:51:10.807416 : CRFOCI:2608355072: crfoci_checkerr: Error - OCI_ERROR ORA-12514: TNS:listener does not currently know of service requested in connect descriptor Now, we have more details related to the connection error. It looks like, ologgerd service is not able to establish a database connection to the management repository as the listener is not aware of the service_name (my_rac_cluster) used to establish the connection. Oracle uses the management repository pluggable database name (named after the cluster_name) as the service name for establishing repository database connection, this can be verified by viewing the repository database configuration as shown below. ---// ---// checking repository database configuration //--- ---// myracserver2 {/home/oracle}: srvctl config mgmtdb Database unique name: _mgmtdb Database name: Oracle home: Oracle user: oracle Spfile: /data/clusterfiles/_mgmtdb/spfile-MGMTDB.ora Password file: Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Type: Management PDB name: my_rac_cluster ---> repository pluggable database name PDB service: my_rac_cluster ---> service_name used for database connection Cluster name: my-rac-cluster Database instance: -MGMTDB We need to make sure that the PDB service (my_rac_cluster) is up and running and is registered with repository listener (MGMTLSNR). Let’s check if the service is registered with the management listener ---// ---// check if PDB service is registered with MGMTLSNR //--- ---// myracserver1 {/home/oracle}: lsnrctl status MGMTLSNR | grep -i MY_RAC_CLUSTER myracserver1 {/home/oracle}: Looks like the PDB service is not registered with the management listener MGMTLSNR, due to which ologgerd service was not able to establish a connection with the management repository database. We can also query the management repository database, to check if the service exist in the database as shown below. ---// ---// checking repository database for PDB service existance //--- ---// SQL> show con_name CON_NAME ------------------------------ CDB$ROOT SQL> show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 MY_RAC_CLUSTER READ WRITE NO ---// ---// switch to repository database //--- ---// SQL> alter session set container=MY_RAC_CLUSTER; Session altered. ---// ---// check if the PDB service exists //--- ---// SQL> select name from dba_services; no rows selected Looks like the PDB service doesn’t exist in the repository database. Lets create the missing PDB service in the repository database as shown below. Note: In general the PDB service is created by default when we install the Clusterware stack. However, if we relocate the repository to its own dedicated location manually, there is a possibility that the PDB service will not be created by default. ---// ---// make sure we are logged in to the repository pluggable database //--- ---// SQL> show con_name CON_NAME ------------------------------ MY_RAC_CLUSTER ---// ---// create the missing PDB service //--- ---// SQL> exec dbms_service.create_service('my_rac_cluster','my_rac_cluster'); PL/SQL procedure successfully completed. SQL> select name from dba_services; NAME ---------------------------------------------------------------- my_rac_cluster ---// ---// start the PDB service //--- ---// SQL> exec dbms_service.start_service('my_rac_cluster'); PL/SQL procedure successfully completed. We have created and started the missing PDB service. Let’s validate if the service is registered with management listener MGMTLSNR. ---// ---// check if PDB service is now registered with MGMTLSNR //--- ---// myracserver1 {/home/oracle}: lsnrctl status MGMTLSNR | grep -i MY_RAC_CLUSTER Service "my_rac_cluster" has 1 instance(s). myracserver1 {/home/oracle}: PDB service is now started and is registered with the management listener. Now, we should be able to query the repository database. Let’s check if can query the repository database ---// ---// validate if repository can be queried //--- ---// myracserver1 {/home/oracle}: $GRID_HOME/bin/oclumon manage -get reppath Connection Error. Could not get RepPath. We are still receiving the same error. Let’s again check the ologgerd service trace file to find out more details about this error. ---// ---// viewing additional error details from ologgerd.trc //--- ---// myracserver2 {/app/oracle/diag/crs/myracserver2/crs/trace}: 2016-02-26 10:36:35.375007 : CRFOCI:2608355072: crfoci_conn_create: OCISessionGetfailed 2016-02-26 10:36:35.440982 : default:2608355072: crflogdbora_getdbconn:crfoci_conn_create failed 2016-02-26 10:36:35.440991 : default:2608355072: crflogdbora_getdbconn: errbuf ORA-28002: the password will expire within 7 days 2016-02-26 10:36:35.440999 : CRFLOGD:2608355072: crflogdb_getRepPath: Unable to connect to managment database 2016-02-26 10:36:35.441002 : CRFLOGD:2608355072: crflogdb_getRepPath: Error : 2016-02-26 10:36:35.441008 : CRFLOGD:2608355072: crflogui_domanage: get Rep Path Failed We have different issue now. This one is related to repository user’s password expiration. By default repository user’s password expiry period is 180 days and is managed automatically by the ologgerd service, which monitors the password expiration and reset it once it is expired. In our case, the password is not yet expired and Oracle is warning about the password state. We can either wait for ologgerd to reset the password automatically or we can manually reset the password using the mgmtca tool as shown below. ---// ---// resetting management repository password //--- ---// myracserver1 {/home/oracle}: $GRID_HOME/bin/mgmtca Let’s check if we are able to query the management repository now. ---// ---// querying management repository //--- ---// myracserver1 {/home/oracle}: $GRID_HOME/bin/oclumon manage -get reppath CHM Repository Path = /data/clusterfiles/_MGMTDB/datafile/o1_mf_sysmgmtd__374325064041_.dbf myracserver1 {/home/oracle}: Yes, we are now able to query the repository database. Problem is resolved !! Footnote: If we are receiving connection error while querying management repository, the first things to check are the repository PDB service (which is used to establish the connection with repository database) status and the repository user’s password (ideally managed automatically by cluster logger service) status and obviously make sure MGMTDB and MGMTLSNR are up and running. Hope you find this information useful.
↧