BusinessObjects system crashes, Java related?

[keithfisherdsc]

Hi,

I have a client with a BOBJ BI4.2 SP7 patch 5 system, on one Windows 2012 R2 server with 24Gb RAM and 8 cores at 2.2GHz. The CMS database is SQL Anywhere 17, which is on the same server as the application.

It is a well-used system, with reports running into the millions of rows of data running relatively regularly on schedules, and users logging in and interacting with live reports. Perhaps a dozen users in it on average.

Starting three weeks ago we have seen the system develop an instability and crash. On the Sunday the system crashes once in the afternoon and needs a manual restart, then on Monday the system has crashed a few times early in the morning and couple of times in the afternoon. It generally stays stable for the rest of the week. These days do see high volumes of data flowing so it could be related.

In the event viewer we always see this pattern: first all 14 connections to the CMS database are lost, one after the other, then the CMS goes into waiting for resources mode, then there are continuing server registration warnings about services not being found in the cluster until the system is rebooted.

At the same time in the logging directory several *dump files are created complaining about a lack of memory for Java processes.

In the CMS trc file this message coincides with the last CMS db connection failing: IDS_WAS_SHUTDOWN_IN_PROGRESS : Error in GetOCAServersByKeySAP BusinessObjects BI platform CMS: CMS system database "BI4_CMS_DSN" is not available. The error cannot be rectified by the end user. Please report this error to your system administrator or database administrator. (FWB 00024) No database connections remaining.

I have witnessed these crashes happening while I have been on the server and seen that the SQL Anywhere service does not fail (no restart of it is required before a SIA restart fixes the problem), and the 24GB physical RAM limit is not approached at any point.

Can anyone think of any causes for the failing connections to the db which might coincide with high usage or some other system process?

thanks

Keith

[DellSC]

What type of reports are you running - Webi, Crystal, or something else?

How much disk space is available on the drive where BOBJ is installed?

[keithfisherdsc]

dell.stinnett-christy - the system runs webi reports only. There is 40GB disc space free.

thanks

Keith

[denis_konovalov · Accepted solution]

the descriptions leads me to one conclusion - it is not properly sized system.
If CMS server not able to connect to CMS DB and crashes -- either the DB is not able to handle all requests or is not able to maintain all connections for some reason.

Neither CMS , nor SQL anywhere are java processes, so, if you see java dumps, then you have more than one issue, but all of them are most likely sizing related.
Sizing is more than CPU/Memory. (recommended min memory size for productive system is 32gb, by the way).

So, to resolve the problem you'll need :
1. run a proper sizing exercise.
2. Troubleshoot CMS server and DB issues.
3. Troubleshoot Java issues, by determining which exact java processes are failing. It could be Tomcat related, Adaptive job servers, Adaptive processing servers etc.

[keithfisherdsc · Accepted solution]

Hi Denis - the Java dumps and the CMS db connection drops happen at exactly the same time (log file timestamps and event viewer timestamps) so they are definitely related/caused by the same thing.

It is normally the scheduling service (AJS) or the WACS that generate the dumps at the time of the crashes. I haven't been able to tie the dumps to the running of particular reports though. There are reports that generate 50million+ rows on this system that run though ok earlier in the day before the crashes (system runs for hours ok afterwards).

Tomcat memory doesn't seem to be a problem.

Got any tips on finding out why the db connections drop - SQL Anywhere doesn't generate any logging files (apart from db logs) that I can find.

thanks

Keith

[denis_konovalov · Accepted solution]

If C++ and Java processes crash at the same time -- that's resource issue.
If AJS crashes - than its related to scheduling.
I'm not sure how are you using WACS, so can't say why would it crash.

First thing I would do is move CMS DB to its own server.