Cluster not working

[#1]

Hi Guys,

We have SAP CRM 7.0 on RHEL 5.3 with oracle as db and clustering defined at OS level. We are facing some problem in fail-over situations, for clustering we have 2 nodes CRMPRD-CI and CRMPRD-DB but both application and db runs on one node, CRMPRD-DB and other node acts as a dead node, we have tried to manuallly switch the application but it fails, I am paisting below the logs of /var/log/messages for your reference:

Apr 21 01:17:38 CRMPRD-CI clurgmgrd[10174]: <notice> Resource Group Manager Starting

Apr 21 01:17:39 CRMPRD-CI bash: [10657]: <warning> sapstartsrv is not running for instance CRP-DVEBMGS01, it will be started now

Apr 21 01:17:40 CRMPRD-CI bash: [10657]: <err> SAP Instance CRP-DVEBMGS01 stop failed: 21.04.2011 01:17:40 Stop FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

Apr 21 01:17:41 CRMPRD-CI SAPCRP_01[11295]: SAP Service SAPCRP_01 successfully started.

Apr 21 01:17:41 CRMPRD-CI bash: [10657]: <warning> sapstartsrv is not running for instance CRP-ASCS00, it will be started now

Apr 21 01:17:41 CRMPRD-CI bash: [10657]: <err> SAP Instance CRP-ASCS00 stop failed: 21.04.2011 01:17:41 Stop FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

Apr 21 01:17:42 CRMPRD-CI SAPCRP_01[11295]: sapstartsrv stopped

Apr 21 01:17:42 CRMPRD-CI SAPCRP_00[11584]: sapstartsrv stopped

Apr 21 01:17:43 CRMPRD-CI clurgmgrd: [10174]: <err> script:CRM-DB: stop of /etc/init.d/CRM-SAPDB failed (returned 1)

Apr 21 01:17:43 CRMPRD-CI clurgmgrd[10174]: <notice> stop on script "CRM-DB" returned 1 (generic error)

Apr 21 01:17:59 CRMPRD-CI clurgmgrd[10174]: <notice> Starting stopped service service:CRMJAVA-CI

Apr 21 01:17:59 CRMPRD-CI avahi-daemon[9915]: Registering new address record for 172.16.4.97 on bond0.

Apr 21 01:18:01 CRMPRD-CI kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory

Apr 21 01:18:01 CRMPRD-CI kernel: NFSD: starting 90-second grace period

Apr 21 01:18:01 CRMPRD-CI clurgmgrd[10174]: <notice> Service service:CRMJAVA-CI started

Apr 21 02:18:12 CRMPRD-CI clurgmgrd[10174]: <notice> Recovering failed service service:CRM-CI

Apr 21 02:18:12 CRMPRD-CI avahi-daemon[9915]: Registering new address record for 172.16.4.112 on bond0.

Apr 21 02:18:14 CRMPRD-CI SAPCRP_00[18270]: SAP Service SAPCRP_00 successfully started.

Apr 21 02:18:15 CRMPRD-CI bash: [17947]: <err> SAP Instance CRP-ASCS00 start failed: 21.04.2011 02:18:15 Start FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Apr 21 02:18:16 CRMPRD-CI SAPCRP_01[18514]: SAP Service SAPCRP_01 successfully started.

Apr 21 02:18:16 CRMPRD-CI bash: [17947]: <warning> sapstartsrv is not running for instance CRP-DVEBMGS01, it will be started now

Apr 21 02:18:16 CRMPRD-CI bash: [17947]: <err> SAP Instance CRP-DVEBMGS01 start failed: 21.04.2011 02:18:16 Start FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

Apr 21 02:18:16 CRMPRD-CI clurgmgrd: [10174]: <err> script:CRM-CI: start of /etc/init.d/CRM-SAP failed (returned 1)

Apr 21 02:18:16 CRMPRD-CI clurgmgrd[10174]: <notice> start on script "CRM-CI" returned 1 (generic error)

Apr 21 02:18:16 CRMPRD-CI clurgmgrd[10174]: <warning> #68: Failed to start service:CRM-CI; return value: 1

Apr 21 02:18:16 CRMPRD-CI clurgmgrd[10174]: <notice> Stopping service service:CRM-CI

Apr 21 02:18:16 CRMPRD-CI SAPCRP_01[18863]: SAP Service SAPCRP_01 successfully started.

Apr 21 02:18:16 CRMPRD-CI bash: [18888]: <warning> sapstartsrv is not running for instance CRP-DVEBMGS01, it will be started now

Apr 21 02:18:16 CRMPRD-CI bash: [18888]: <err> SAP Instance CRP-DVEBMGS01 stop failed: 21.04.2011 02:18:16 Stop FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

Apr 21 02:18:17 CRMPRD-CI bash: [18888]: <err> SAP Instance CRP-ASCS00 stop failed: 21.04.2011 02:18:17 Stop FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Apr 21 02:18:17 CRMPRD-CI SAPCRP_01[19157]: SAP Service SAPCRP_01 successfully started.

Apr 21 02:18:17 CRMPRD-CI SAPCRP_00[18270]: sapstartsrv stopped

Apr 21 02:18:17 CRMPRD-CI SAPCRP_01[19157]: sapstartsrv stopped

Apr 21 02:18:17 CRMPRD-CI avahi-daemon[9915]: Withdrawing address record for 172.16.4.112 on bond0.

Apr 21 02:18:27 CRMPRD-CI clurgmgrd[10174]: <notice> Service service:CRM-CI is recovering

Apr 21 02:18:27 CRMPRD-CI clurgmgrd[10174]: <notice> Starting stopped service service:CRM-CI

Apr 21 02:18:27 CRMPRD-CI avahi-daemon[9915]: Registering new address record for 172.16.4.112 on bond0.

Apr 21 02:18:29 CRMPRD-CI SAPCRP_00[19988]: SAP Service SAPCRP_00 successfully started.

Apr 21 02:18:29 CRMPRD-CI bash: [19731]: <err> SAP Instance CRP-ASCS00 start failed: 21.04.2011 02:18:29 Start FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Apr 21 02:18:29 CRMPRD-CI SAPCRP_01[20223]: SAP Service SAPCRP_01 successfully started.

Apr 21 02:18:29 CRMPRD-CI bash: [19731]: <warning> sapstartsrv is not running for instance CRP-DVEBMGS01, it will be started now

Apr 21 02:18:29 CRMPRD-CI bash: [19731]: <err> SAP Instance CRP-DVEBMGS01 start failed: 21.04.2011 02:18:29 Start FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

Apr 21 02:18:29 CRMPRD-CI clurgmgrd: [10174]: <err> script:CRM-CI: start of /etc/init.d/CRM-SAP failed (returned 1)

Apr 21 02:18:29 CRMPRD-CI clurgmgrd[10174]: <notice> start on script "CRM-CI" returned 1 (generic error)

Apr 21 02:18:29 CRMPRD-CI clurgmgrd[10174]: <warning> #68: Failed to start service:CRM-CI; return value: 1

Apr 21 02:18:29 CRMPRD-CI SAPCRP_01[20532]: SAP Service SAPCRP_01 successfully started.

Apr 21 02:18:29 CRMPRD-CI clurgmgrd[10174]: <notice> Stopping service service:CRM-CI

Apr 21 02:18:30 CRMPRD-CI bash: [20559]: <err> SAP Instance CRP-ASCS00 stop failed: 21.04.2011 02:18:30 Stop FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Apr 21 02:18:30 CRMPRD-CI SAPCRP_00[19988]: sapstartsrv stopped

Apr 21 02:18:30 CRMPRD-CI SAPCRP_01[20532]: sapstartsrv stopped

Apr 21 02:18:30 CRMPRD-CI avahi-daemon[9915]: Withdrawing address record for 172.16.4.112 on bond0.

Apr 21 02:18:40 CRMPRD-CI clurgmgrd[10174]: <notice> Service service:CRM-CI is recovering

Apr 21 04:02:04 CRMPRD-CI pidof[13121]: can't read sid from /proc/13101/stat

Apr 21 15:58:16 CRMPRD-CI pidof[10502]: can't get program name from /proc/10504/stat

Please advise.

REgards,

Mridul

[Former Member]

I didn't fully read the log because its formatting is messed up. Please attach it as a text file.

I just saw that the NI connect didn't work which points to the network. Did you adapt /etc/services on the second node?

[former_member189546]

hello,

Do you get a corresponding error in windows event viewer

From command prompt run sapstartsrv -v do you get an error.

regards,

John Feely