SAP business Objects following upgrade, unable to login

[Luke_D]

Environment:

• Current: SAP BusinessObjects BI Platform 4.3 Support Pack 3 Patch 3 (Clustered environment with ALB on SUSE linux) 
• Upgrade: SAP BusinessObjects BI Platform 4.3 Support Pack 4 Patch 2

Question:

What would cause an upgrade to be "Successful" but attempts to authenticate via CMC or Launchpad following upgrade are "Delayed" to the point of being unsuccessful (but no error is presented after waiting several minutes) using LDAP or Enterprise authentication (the two supported choices for the environment.)?

Details: 

Environment was successfully upgraded in dev/test without issue: and production shows successful as well; however... 

• Following upgrade, attempts to authenticate fail.  Admin group user attempts to authenticate in CMC or in BI Launchpad using either LDAP or Enterprise and both "never login"  they just sit idle with no change in presentation screen.
• As part of upgrade: BI launchpad, Web Intelligence, SAP Analytics Cloud and RESTful web services had no access set for everyone group to prevent login.  Admins however should continue to be able to login; (they could in past upgrades)
• The page failing when using developer tools network is logon.jsp  it's stuck waiting for server response and never returns.
• The CMS page logon.faces has similar behavior.
• Prior to upgrade 1 out of 5 logins would be "Delayed" for around 2 minutes; but would always let the users in. This was considered a separate issue and is being looked into. 
• Using Test environment URL(TOMCAT) , we pointed system to production cluster and experienced the "no logon no error"
• Using the Production environment URL (TOMCAT)  we pointed system to "TEST" and log-in proceeded as expected.
• Direct connection to the CMS worked; and we were able to issue queries.

Updated with Additional Details:

• We did try connecting to each CMC server directly bypassing the ALM.  This approach exhibited the same behavior.  Enter admin level credentials; wait and wait and wait and nothing happens. It sits on the login.js page in Launchpad indefinitely We also tried this in CMC to the same results. When we use TOMCAT production to connect to Test environment it works.  So, the thought is TOMCAT is working as expected.  We connected to the CMS Database via a local logon to the database and were able to issue queries and obtain results.  So, the CMS database is responding to requests.   When we issue network commands from the TOMCAT server to the CMS database to ensure network connectivity isn't a problem we are able to connect from TOMCAT to the CMS database on the required ports.  Ping is also responsive; so there doesn't appear to be a firewall issue (Remember it was working before the upgrade so network/firewall/AV/Monitoring services all seem like they wouldn't be causing this)
• SAP recommended running a repair tool.  When this was tried, everything completed successfully; but had no impact on our situation. 

[Luke_D · Accepted solution]

We were finally successful. 

At first we just kept disabling services and restarting SIA until we were able to consistently login.  Doing so we isolated it down to the server running "Core" services.

• Insight to Action
• Monitoring
• Publishing Post Processing,
• Publishing Service
• Security Token Service
• Trace Log Service
• Translation Service

We then left this disabled create a new server and added the individual components until we found that the "Monitoring Service" was the one causing problems.  

From there we learned that under CMC-->applications-->Alerting Application, Unchecking email notifications allowed us to keep the monitoring service running; but with out email notifications. 

This made us ask why: so we dug into alerts/watches using email notifications from  BI Admin Studio.   under the: watches; there were several system generated watches which we could not get to step "Review" when editing.  This lead us to an assumption corrupt watches existed and email were somehow preventing/causing CMS to become unresponsive. 

Once we disabled the email alerts; the login issue went away completely.  Following the upgrade we deleted and re-created the services which had corrupt Watches.  (those watches which wouldn't let us get through all 4 steps + review when editing.  Now all watches can get through the "Review" step when editing.  We'll see if that helps next time; but we also presently have left off off the email alerts from Applications-->Alerting Applications within CMC. 

As to why Watches alerts would prevent login... We have no idea.  Somehow the corrupt alerts where causing the server to spin, deadlock and subsequent calls to the CMS database were hung up.  If we cleared a lock; then login would work; but there was no deadlock; just a DB waiting on the CMS server to "Finish" a transaction which it never did.... 

[Kishan_D]

Hello,

Looking at the described behavior this might be a case of Tomcat hanging when trying to get the service to discover available CMS to process the logon. I might suggest trying login bypassing the ALB and using direct Tomcat URL - if already tried, let us know if this works. 

If the same behavior is seen via ALB or direct Tomcat URL, further investigation of traces is required as per SAP Note 3042966. The gathered thread dumps can provide more information on the issue cause.

I would also suggest checking SAP Note 2468820 and related notes - to ensure firewall rules across the cluster nodes do not cause login issues.

Regards, Kishan

[Luke_D]

Updated main note: but yes we tired direct URL of the CMS servers and IP's. This was met with same behavior of no error, but no login. I'm told there are no firewalls which would block login as telnet to relevant IP:Ports are responsive. We've had to Rollback to our prior version and everything is back and running; so I am unable to get additional information at this time. We did take an image of the servers involved and will be restoring those in isolation as we attempt to isolate the cause of this. We are also working with SAP on a ticket to see what can be done. Just looking for threads we may not have considered. Appreciate the feedback!