Question Re: CCMS Agents availability in Technology Q&A

CCMS Agents availability

Former Member — Wed, 11 Nov 2015 10:20:07 GMT

Hi All,

ccms agent availability is showing 0% after rebooting the remote system for winter time change.

agent connection test is successful and work directory is accessible in rz21. looks like agent is not able to talk to central system. any troubleshooting steps?

the error in "sapstartsrv.log" is

CCMS agent initialization for instance type ABAP: return code 0.

CCMS agent start: return code 411.

Thanks,

Ranjith

Re: CCMS Agents availability

alwina_enns — Wed, 11 Nov 2015 13:32:19 GMT

Hello Ranjith,

could you please check the agent trace sapstartsrv_ccms.log in DIR_LOGGING/sapccm4x directory of the affected instance, if any errors are reported for the CSMREG user and if this user could connect to CEN?

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Wed, 11 Nov 2015 13:49:23 GMT

Hello Alwina,

I reregistered the agents and availability became 100% and then dropped to 0%. then again it became 33% and drops to 0%.

sapstartsrv_ccms.log has below errors.

ERROR: OS monitoring: Error-253 Can not attach same MTE [Utilization] with different MTE class [CPU_UsageUtil]. Existing MTE class: [CPU_Utilization]

ERROR: OS monitoring: Error-253 Can not attach same MTE [I/O Wait] with different MTE class [CPU_UsageIOWait]. Existing MTE class: [CPU_IO_Wait]

ERROR: OS monitoring: Error-253 Can not create MTE [Solution Version]. Same name but different types MESSAGE_CONTAINER vs. SHORTTEXT

sapstartsrv.log has error for ccms

CCMS agent initialization for instance type ABAP: return code 0.

CCMS agent start: return code 411.

Thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Wed, 11 Nov 2015 14:02:00 GMT

Hello Ranjith,

in which client you was logged on during the agent registration? The registration of the integrated agents is client dependent.

What is the basis support package level of CEN? As of certain basis support package level a new component registration was delivered in RZ21.

What is the kernel pach level of the monitored system?

In the agent trace please search for CSMREG user. The errors above error-253 indicates some inconsistency in the OS monitoring, but these errors are not the cause of the issue. If the CSMREG user could connect to CEN without problems, you should see an entry like this:

Re: CCMS Agents availability

Former Member — Wed, 11 Nov 2015 15:20:42 GMT

Hi Alwina,

i registered in client 000 and basis is sp15 SAPKB70215. kernel patch of monitored system is 721_ext patch 500.

i tried searching csmreg in trace file but nothing came up

hbpadm> grep CSMREG sapstartsrv_ccms.log

hbpadm>

thanks,

Ranjith

Re: CCMS Agents availability

alwina_enns — Wed, 11 Nov 2015 19:14:16 GMT

Hello Ranjith,

thank you, SAPKB70215 contains the old component registration in RZ21. If sapstartsrv_ccms.log does not contain any entries about CSMREG user, then it is probably too big because of too many entries with the Error-253 and you should check, if sapstartsrv_ccms.log was already moved to sapstartsrv_ccms_1st.log. If such logs exist, please search in all sapstartsrv_ccms* files for CSMREG user. You should find either an error for CSMREG or the info, that CSMREG is connected to CEN.

Did you get any errors or messages during the system registration in RZ21?

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Thu, 12 Nov 2015 08:22:32 GMT

Hi Alwina,

there are no messages related to csmreg in any of the files. i did not get any error during reregistration. wonder why the error is coming after restart on Nov 1. csmreg initial password expired and i had to reset it before re-registering the agents.

thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Thu, 12 Nov 2015 08:53:03 GMT

Hello Ranjith,

if you have changed the password for the CSMREG user, then this is possible, that the old password is still stored somewhere, either in CEN or in the working directory of the agent, and the reregistration should help. Could you please:

1. Unregister the agent again by deleting it in "RZ21 --> Topology --> Agents for Remote Systems --> Display Overview"

2. On the monitored system please execute one time manually:

sapccm4x -initshm pf=<full path to the profile used by the integrated agent>

(please look in the first trace sapstartsrv_ccms_1st.log for the first entries, which profile is used)

This will reinitilize the entries of the agent in the CCMS shared memory

3. Please move the whole content of the agent working dircetory DIR_LOGGING/sapccm4x to another place

4. Restart sapstartsrv of the affected monitored instance with:

sapcontrol -nr NN -funciton RestartService

("NN" - instance number)

5. Logon in client 000 in CEN and register the system in RZ21 again

6. If the CCMS in CEN is configured in a different client than 000, you will need to repeat the system registration also in this client

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Thu, 12 Nov 2015 09:07:15 GMT

Hi Alwina,

i will perform these steps and update you. I cannot restart sapstartsrv as it is PRD system. i am also seeing this error in sapstartsrv. normally we should have return code 0 . isnt it?

the error in "sapstartsrv.log" is

CCMS agent initialization for instance type ABAP: return code 0.

CCMS agent start: return code 411.

thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Thu, 12 Nov 2015 09:28:08 GMT

Hello Ranjith,

yes, the return code for the agent start should be 0, I do not know, what the return code 411 means, but you should see more details in the agent trace, what was the problem during the start.

The sapstartsrv can be restarted on OS level with sapcontrol as above at any time, it does not restart the whole system, you can try it out on a test system first. It could be only a problem, if the affected instance is running in a cluster, you should clarify first, how cluster software will react, if sapstartsrv is not available for a short time.

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Thu, 12 Nov 2015 12:31:39 GMT

Hi Alwina,

I performed all the steps you gave and now the error during reregistration is

Error when sending the configuration data to the agent

SoapFaultCode:5 CCMS__RegisterCentralSystem failed: 413: the agent is restarting now, please resend your request latter.

Thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Thu, 12 Nov 2015 12:38:56 GMT

Hello Ranjith,

please check the agent trace sapstartsrv_ccms.log, what is the problem with the agent?

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Thu, 12 Nov 2015 13:05:35 GMT

Hello Alwina,

this is the log in one of the server where sapcontrol restart gave me error

sapcontrol -nr 02 -function RestartService

12.11.2015 07:43:49

RestartService

FAIL: HTTP error, HTTP/1.1 401 Unauthorized

> more sapstartsrv_ccms.log

[Thr 139760988591968]

Thu Nov 12 07:08:55 2015

Initialize CCMS agent sapccm4x

INFO:

CCMS version 20060508, 64 bit, multithreaded, Unicode 7210

compiled at Apr 27 2015

systemid 390 (AMD/Intel x86_64 with Linux)

running on Linux 2.6.32-573.1.1.el6.x86_64 #1 SMP Tue Jul 14 02:46:51 EDT 2015 x86_64

patch text patch collection 2014/1, OSS note 1453112, PL 500

profile /usr/sap//SYS/profile/

INFO: cannot open config file /usr/sap//D02/log/sapccm4x/csmconf_sapstartsrv.

INFO: CCMS agent is not registered at central CCMS system.

INFO: No central CCMS system configured.

INFO: CCMS agent initialization finished.

INFO: Starting CCMS agent threads...

INFO: Checking CCMS monitoring segment ...

INFO: Attached to a monitoring segment type APPLSERVER (1), running in shared memory.

INFO: Agent status in the attached monitoring segment is ACTIVE (20).

ERROR: Starting integrated CCMS agent SAPCCM4X encountered troubles when attaching CCMS monitoring segment of type APPLSERVER:

There is an agent, pid 29069, with the same profile already active. Cannot start twice!

If there is no other standalone or integrated CCMS agent SAPCCM4X running with a profile containing

parameters "SAPSYSTEMNAME = " and "SAPSYSTEM = 2", start standalone agent sapccm4x with option

-initshm and profile to perform cleanup of CCMS Monitoring Segment in shared memory.

INFO: Checking CCMS monitoring segment finished.

****************************

* CCMS AGENT START ABORTED *

thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Thu, 12 Nov 2015 13:16:20 GMT

Hello Ranjith,

thnak you for the info. Under which user you are logged on to restart the sapstartsrv? The <sid>adm user should not get an authorization error.

It looks like there is an agent.lock file in the working dircetory of the agent? The "initshm" option should delete this file. Please execute the agent with the "initshm" option again under the <sid>adm user:

sapccm4x -initshm pf=<path to the profile used by the integrated agent>

and restart sapstartsrv (under <sid>adm user) and check the sapstartsrv_ccms.log trace, if the agent could start.

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Thu, 12 Nov 2015 13:24:32 GMT

Hello Alwina,

i did try the steps again but it didnt help. .sapstream5xx13 is not getting created in /tmp directory . i might have to restart the entire instance.

thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Thu, 12 Nov 2015 13:47:00 GMT

Hello Ranjith,

is this still the same authorization error also for the <sid>adm user? Or which error did you get?

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Thu, 12 Nov 2015 13:50:34 GMT

Hello Alwina,

it is same error for sidadm

thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Thu, 12 Nov 2015 14:25:27 GMT

Hello Ranjith,

did you check already the note "1565645 - SAP composite note:sapcontrol" for the reasons of the "Unauthorized" error?

Could you please execute sapcontrol with the <sid>adm user and password:

sapcontrol -nr NN -user <sid>adm <password> -function RestartService

Does it work?

You have mentioned, that the file .sapstream5xx13 is not getting created. What/who has removed this file?

Regards,
Alwina

Re: CCMS Agents availability

Former Member — Fri, 13 Nov 2015 13:43:45 GMT

Hi Alwina,

I tried sapcontrol -nr NN -user <sid>adm <password> -function RestartService but syntax seems wrong. please share help link for sapcontrol commands syntax?

thanks,
Ranjith

Re: CCMS Agents availability

alwina_enns — Fri, 13 Nov 2015 13:55:57 GMT

Hello Ranjith,

on our test system it works: