Linux - dbsrv16 terminates unexpectedly

[Former Member]

I have a linux machine that runs 50 instances of the same 50Mb database.

cmd for each looks like this:

dbsrv16 -m -ud -c 64M -ch 256M "-x TCPIP(port=2001)" "-xs HTTP(port=2501)" -ti 0 -tl 0 -on 10m -oe /peopleweb/db*NN*/errors.txt -o /peopleweb/db*NN*/console.txt -n pw*NN* /peopleweb/db*NN*/PW16_pw.db


With NN starting at 01 up to 50
These databases are presently all just idling.  No workload.  But between 2-5 hours, a random one would just stop running.  Upon inspecting the /var/log folder in my Ubuntu server, I came across the following in the apport.log file:
ERROR: apport (pid 4949) Fri Feb  6 03:40:08 2015: called for pid 2985, signal 6, core limit 0

ERROR: apport (pid 4949) Fri Feb  6 03:40:08 2015: executable: /opt/sqlanywhere16/bin64/dbsrv16     (command line "dbsrv16 -m -ud -c 64M -ch 256M -x\\ TCPIP(port=2029) -xs\\ HTTP(port=2529) -ti 0 -tl 0 -on 10m -oe /peopleweb/db29/errors.txt -o /peopleweb/db29/console.txt -n pw29 /peopleweb/db29/PW16_pw.db")

ERROR: apport (pid 4949) Fri Feb  6 03:40:08 2015: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment

ERROR: apport (pid 4949) Fri Feb  6 03:40:08 2015: apport: report /var/crash/_opt_sqlanywhere16_bin64_dbsrv16.1002.crash already exists and unseen, doing nothing to avoid disk usage DoS


Any idea what could cause this, or where I should start looking?
What is the maximum number of databases I can run on a single machine?
I don't want to run a single server with multiple databases as each should listen on a different port.

[VolkerBarth]

What is the maximum number of databases I can run on a single machine?

To cite a Jeff Albion answer from a comment on that other FAQ:

Q: Is there a technical limit as to the number of database engines on one physical server, i.e. could one machine run 200 database engines each running 250 databases thus running 5000 databases in total?

A: No, you're only limited by the total OS memory constraints for all of the database servers in this scenario. Eventually you will get into a situation where we can't allocate all of the database start-up cache/worker threads (due to lack of resources) on one of the database servers.

I can't tell on the reported error, however, running 50 instances will certainly have a deep impact on resource using/sharing compared to one instance with 50 databases in total as the latter would apparently share resources between the databases "by design". Nevertheless, as you say, that would not allow different TCP/IP ports for each database.

[Breck_Carter]

I assume there's nothing helpful in either diagnostic log file -oe /peopleweb/dbNN/errors.txt -o /peopleweb/dbNN/console.txt

I also assume you know the implications of -m (and it probably has nothing to do with this symptom)

Generally speaking, -tl 0 is a very bad idea (unlike -ti 0 which is often a requirement). The -tl 0 option essentially tells the server to never drop dead/defunct connections, and I've never seen a situation where that is desirable... but again, it probably has nothing to do with this symptom.

When you say "stop running" do you mean "crashes" or do you mean "stops responding to requests"?

You may want to call SAP Tech Support... AFAIK there's not a lot of Ubuntu apport debugging expertise on this forum.

[Former Member]

Thank you Breck. The -tl0 is a relic from our Client/Server application. Will take that out.

The database simply crashes. Dies. Nothing in the two files either.

Thanks for the advice.

[jeff_albion]

called for pid 2985, signal 6, core limit 0

This is indicating that the database server process (pid 2985) received a signal 6 (SIGABRT). SQL Anywhere will typically issue a SIGABRT during a database assertion. (See KBA 1958942 - Error: "Assertion failed: 123456" / SQLCODE -301 "Internal Database Error" - Handling an Assertion Failure)

Any idea what could cause this, or where I should start looking? ... Nothing in the two files either.

Just to confirm the scenario: so for two hours, the database server was running, checkpointing, etc. and then crashed with no message in the console log (/peopleweb/dbNN/console.txt) or error log (/peopleweb/dbNN/errors.txt)? We will likely need to take a look at the core file for the crashing process in SAP Support to understand what's happening in this situation further.

---

Typically when SQL Anywhere crashes, we have a signal handler that attempts to write out a core file to the current working directory of the database server (using the default core_pattern). However, since you are using -ud (or if you had used dbspawn), we are following standard UNIX daemon methodology by setting the current working directory to root "/", so no core file will be created there (likely due to permissions that the process is running under).

Under these situations to generate a core file you must either:

• Prevent the database servers from running as a daemon (do not use -ud or do not use dbspawn) and run them at the command line instead
• Run the server as a user which has permissions to write to the root directory (not advisable, but perhaps okay in a contained test environment)
• Change the configuration file /proc/sys/kernel/core_pattern on Linux to cause the core files to be written into a different directory.

In this case (Ubuntu), core_pattern defaults to |/usr/share/apport/apport %p %s %c, which passes the core off to apport to generate. You will likely need to enable and configure apport to store the core files correctly so we can examine them later. Typically core files from apport are stored in /var/crash/ and you may have to configure RLIMIT_CORE.

---

For more information about SQL Anywhere and core files, and to understand which files we will need to collect in technical support to analyze the core files for more information, see the following KBAs:

• KBA 2075516 - What to do when encountering a SQL Anywhere database crash (core file) on Linux
• KBA 2075551 - How To Collect Core Files on Linux for SQL Anywhere Processes during Hangs and Crashes
• KBA 2114328 - How To Collect Required Linux System Libraries to Analyze Linux Core Files for SQL Anywhere

[Former Member]

Thank you Jeff. Will go through this and revert back to you.

[Former Member]

Jeff

I have located the core files in /var/crash

/var/crash/_opt_sqlanywhere16_bin64_dbsrv16.1002.crash

Is this what you're referring to?

Could this be submitted to SAP for analysis?

[jeff_albion]

Yes, that sounds like the core file we would need to collect, generated from apport.

Could this be submitted to SAP for analysis?

Yes, for sure. But as mentioned above, we will also need to collect the system libraries from the crashing system in order to open the core file for a detailed analysis - can you open an incident with SAP Support?

Also depending on how big the core file is, you may have to upload it to us via FTP - we can provide the details on how to do this via the incident.