Monday, March 7, 2016

ORA-29701: unable to connect to Cluster Synchronization Service

Recently we got this issue on our 12C db .
We started getting error in all applications log as ORA-29701: unable to connect to Cluster Synchronization Service

we have ASM without RAC . DB 12.1.0.2 

What log shows : 


crsctl check has


this errors out unable to communicate to high availability .


restarted machine and issue got resolved .


Root cause : Some one by mistake ran rm -fr on /tmp and  that iuntern removed socket files from /tmp/.oracle issue .

We found details only after followup on sr .



From alert.log
===========
2016-03-01 12:31:28.503 [ORAAGENT(3757)]CRS-5016: Process "/oracle/grid/12.1.0/product/bin/lsnrctl" spawned by agent "ORAAGENT" for action "check" failed: details at "(:CLSN00010:)" in "/oracle/grid/12.1.0/diag/crs/iprdebd1/crs/trace/ohasd_oraagent_oragrid.trc"
2016-03-01 12:31:37.582 [ORAAGENT(3757)]CRS-5020: Not all endpoints are registered for listener LISTENER
2016-03-01 14:04:05.802 [CLSECHO(14981)]CRS-10001: CRS-6021: Could not set system resource limits for Clusterware: "ulimit -n 65536".<<<<<<<<<<<<<<
2016-03-01 14:14:12.247 [OHASD(14978)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 14978
2016-03-01 14:14:12.248 [OHASD(14978)]CRS-0715: Oracle High Availability Service has timed out waiting for init.ohasd to be started.<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-03-01 15:11:26.249 [OHASD(2972)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 2972
2016-03-01 15:11:26.258 [OHASD(2972)]CRS-0714: Oracle Clusterware Release 12.1.0.2.0.
2016-03-01 15:11:26.346 [OHASD(2972)]CRS-2112: The OLR service started on node iprdebd1.
2016-03-01 15:11:26.538 [OHASD(2972)]CRS-1301: Oracle High Availability Service started on node iprdebd1. 
From listener.log 
=================
01-MAR-2016 12:30:45 * service_update * IFFPROD * 0
01-MAR-2016 12:30:51 * service_update * IFFPROD * 0<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<Tue Mar 01 12:31:28 2016
System parameter file is /oracle/grid/12.1.0/product/network/admin/listener.ora<<<<<<<<<<<<<<Log messages written to /oracle/grid/12.1.0/diag/tnslsnr/iprdebd1/listener/alert/log.xml
Trace information written to /oracle/grid/12.1.0/diag/tnslsnr/iprdebd1/listener/trace/ora_7662_1.trc
Trace level is currently 0 

Thanks

No comments:

Post a Comment