SAP HANA High Availability, SLD and 'Failed to connect System Landscape Directory(SLD)'

[giacomozzi]

We are facing a serious problem and we are not finding its cause.

First let me present the scenario:
We have two physical Linux servers running HANA 2.0 with high availability, that is, data from the primary server is replicated on the secondary server. Each server has 196GB of memory and 128GB is licensed for HANA. Peak memory usage is around 80GB.
Then we have a virtual Linux server running the SLD services, with CPU, memory and disk resources apparently left over.

The problem:
From time to time, once every fifteen days, the SAP client crashes and reports the following message: 'Failed to connect System Landscape Directory(SLD);contact your system administrator'. No one can log out or log in to the system and even third party connections are not possible as the 'ERROR [S1000] [SAP AG][LIBODBCHDB DLL][HDBODBC] General error;437 only commands for license handling are allowed in current state ERROR [S1000] [SAP AG][LIBODBCHDB DLL][HDBODBC] General error;-10210 Invalid command state (No prepared SQL command)' error is displayed.

Workaround:

To reestablish access I stop the services on the SLD server:

service b1s stop && /etc/init.d/sldagent stop && /etc/init.d/sapb1servertools stop

Restart the HANA service on the HANA server:

su - hdbadm -c "HDB stop"
su - hdbadm -c "HDB start"

And finally I start the services on the SLD server:

/etc/init.d/sapb1servertools start && /etc/init.d/sldagent start && service b1s start

This way, everything works again, without having to restart the servers.

What we have already tried to do:

The network core switches were replaced;
We changed "listeninterface", which was "local" to "global";
We send several traces to SAP.

SAP Response:

'The problem is network related. 786 matches were found, only in the TRACE files relating to tenant HDB'.

System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannel>={<NetworkChannelBase>={this=140498106888216, fd=53, refCnt=1, local=127.0.0.1/26356_tcp, remote=127.0.0.1/30004_tcp, state=ConnectWait, pending=[-- --]}}}; $Context$=[33872db788d730a7,127.0.0.1:26356,127.0.0.1:30004,TRN,0]

It turns out that the network is working, the servers are still accessible, no other services on the network have problems, and the message in the trace refers to localhost.

Thanks in advance for any help, because this problem has been going on for months.

Thanks!

[giacomozzi]

There appears to be a problem with the license.
We have already carried out the procedure described at https://me.sap.com/notes/0002832368, but without success. The problem persists.
I have the impression that memory usage increases until the system blocks usage.

[msundararaja_perumal]

giacomozzi,

You may verify if this is caused due to HA, in my experience sometimes the log message from HANA might not be directly related to the actual issue.

Thanks.

[giacomozzi]

I still haven't discovered the cause of the problem. This occurs approximately every 20 days.