SA12 server crash without any reports

[Former Member]

We have been using SA12 for quite sometime now without any problems, yesterday morning we started experiencing server crashes every 30-60 minutes, they continue to happen today. I have applied the latest EBF 12.0.1.3519.

I've read a number of posts on the forum and applied the following to the server startup options: -o "/usr/jobbeat/server_log.txt" -oe "/usr/jobbeat/server_start_log.txt" -os 1M

When the server stops there are no error messages written to the log files, I've also run dbsupport -lc but it tells me there are no unsubmitted reports found, I've looked in the DB files directory and there are no dump files, I've also searched the rest of the system and could not locate any files.

So it looks like the server shuts down without being able to write any log entries or create any dump files. So at this stage I have no idea what is causing the problem and how I can track it down.

Any help greatly appreciated.

Regards,

Etienne.

[Former Member]

Do the system event logs report anything for SQL Anywhere?

Can you post the tail lines of the server log?

---

If this is a critical application, you may want to contact support for direct one-on-one communication to help resolve the issue.

See: http://www.sybase.com/contactus/support/

[Former Member]

Hi Tyson,

Thank you for your response it's much appreciated. Here are the last few lines before the crash and then the output on the restart of the server from the messages log these messages are the same as the one's written to the log file specified using the -o option:

Apr 17 10:17:54 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 10:17
Apr 17 10:18:02 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting checkpoint of "jobbeat" (jobbeat.db) at Tue Apr 17 2012 10:18
Apr 17 10:18:08 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat" (jobbeat.db) at Tue Apr 17 2012 10:18
Apr 17 10:32:49 jobbeat-db syslog-ng[2925]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=366', processed='center(
received)=345', processed='destination(newsnotice)=0', processed='destination(acpid)=2', processed='destination(firewall)=0', processed='destination(null)=2', processed='des
tination(mail)=2', processed='destination(mailinfo)=2', processed='destination(console)=5', processed='destination(newserr)=0', processed='destination(newscrit)=0', processe
d='destination(messages)=339', processed='destination(mailwarn)=0', processed='destination(localmessages)=3', processed='destination(netmgm)=0', processed='destination(maile
rr)=0', processed='destination(xconsole)=5', processed='destination(warn)=6', processed='source(src)=345'
Apr 17 10:37:54 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 10:37
Apr 17 10:37:55 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 10:37
Apr 17 10:55:07 jobbeat-db sudo:     root : TTY=pts/0 ; PWD=/usr/jobbeat ; USER=root ; COMMAND=/bin/rm -f /var/run/SA_jobbeatasa.pid
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 SQL Anywhere Network Server Version 12.0.1.3519
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Workgroup Edition
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Copyright (c) 2001-2011, iAnywhere Solutions, Inc.
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Portions copyright (c) 1988-2011, Sybase, Inc. All rights reserved.
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Use of this software is governed by the Sybase License Agreement.
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Refer to http://www.sybase.com/softwarelicenses.
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Processors detected: 2 (containing 8 logical processors)
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Processor limit (Workgroup Edition😞 2
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Processor limit (licensed processors😞 2
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Maximum number of physical processors the server will use: 2
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 This server is licensed to:
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Bayside
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Bayside Group
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Running Linux 2.6.27.42-0.1-default #1 SMP 2010-01-06 16:07:25 +0100 on X86_64
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Server built for X86_64 processor architecture
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 1291836K of memory used for caching
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Minimum cache size: 13548K, maximum cache size: 9098532K
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Using a maximum page size of 4096 bytes
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Multiprogramming level: minimum:8, current:20, maximum:80
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Automatic tuning of multiprogramming level is enabled
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting database "jobbeat" (/usr/jobbeat/jobbeat.db) at Tue Apr 17 2012 10:55
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting database "jobbeat_archive" (/usr/jobbeat/jobbeat_archive.db) at Tue Apr 17 2012 10:55
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Database recovery in progress
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Last checkpoint at Tue Apr 17 2012 10:18
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Checkpoint log...
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Database recovery in progress
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Last checkpoint at Tue Apr 17 2012 10:37
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Checkpoint log...
Apr 17 10:55:11 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Transaction log: jobbeat_archive.log...
Apr 17 10:55:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Checkpointing...
Apr 17 10:55:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 10:55
Apr 17 10:55:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 10:55
Apr 17 10:55:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Recovery complete
Apr 17 10:55:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Database "jobbeat_archive" (jobbeat_archive.db) started at Tue Apr 17 2012 10:55
Apr 17 10:55:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Transaction log: jobbeat.log...
Apr 17 10:55:17 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     ... recovery 0% complete
Apr 17 10:55:27 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     ... recovery 90% complete
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Rollback log...
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     Checkpointing...
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting checkpoint of "jobbeat" (jobbeat.db) at Tue Apr 17 2012 10:55
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat" (jobbeat.db) at Tue Apr 17 2012 10:55
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Recovery complete
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Database "jobbeat" (jobbeat.db) started at Tue Apr 17 2012 10:55
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Database server started at Tue Apr 17 2012 10:55
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Trying to start SharedMemory link ...
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     SharedMemory link started successfully
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Trying to start TCPIP link ...
Apr 17 10:55:28 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting on port 2638
Apr 17 10:55:33 jobbeat-db SQLAnywhere(jobbeatasa_suse😞     TCPIP link started successfully
Apr 17 10:55:33 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Now accepting requests
Apr 17 10:55:34 jobbeat-db sudo:     root : TTY=pts/0 ; PWD=/usr/jobbeat ; USER=root ; COMMAND=/bin/echo 1189
Apr 17 11:15:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 11:15
Apr 17 11:15:12 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 11:15
Apr 17 11:32:49 jobbeat-db syslog-ng[2925]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=425', processed='center(
received)=404', processed='destination(newsnotice)=0', processed='destination(acpid)=2', processed='destination(firewall)=0', processed='destination(null)=2', processed='des
tination(mail)=2', processed='destination(mailinfo)=2', processed='destination(console)=5', processed='destination(newserr)=0', processed='destination(newscrit)=0', processe
d='destination(messages)=398', processed='destination(mailwarn)=0', processed='destination(localmessages)=3', processed='destination(netmgm)=0', processed='destination(maile
rr)=0', processed='destination(xconsole)=5', processed='destination(warn)=6', processed='source(src)=404'
Apr 17 11:35:13 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Starting checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 11:35
Apr 17 11:35:13 jobbeat-db SQLAnywhere(jobbeatasa_suse😞 Finished checkpoint of "jobbeat_archive" (jobbeat_archive.db) at Tue Apr 17 2012 11:35

[MarkCulp]

The log that you have posted shows that you are using Linux. Since you are not seeing any core files check that you have set core file size to unlimited (i.e. ulimit -c unlimited) before starting your server - Since it appears that you are starting your server as a service you may need to edit the SA_jobbeatasa script in /etc/init.d to do this (i.e. just add "ulimit -c unlimited" to the script before it sources dbsvc).

Since you are seeing the crash fairly regularly I suspect that something has corrupted your database and the server is hitting this corruption every 30-60 minutes in a unexpected way and hence it is not being caught before the server dies. You may want to run dbvalid on the database to check for corruptions?

[Former Member]

Hi Mark,

Thanks you for the suggestions, I will make the change to the service startup script as suggested.

I did a complete unload/reload into new database files last night as I thought there may be a corruption as well. Is this enough or should I still do the dbvalid?

Thanks again.

[MarkCulp]

No need to do a dbvalid on the new database.... unless you continue to see the crash. If you do see the server crash (disappear) on the new database then you might be hitting a different problem and you will likely need to contact Tech Support to help you diagnose the issue.

[Former Member]

It crashed again just now. I did set the ulimit(see below) and still no core files:

jobbeat-db:/usr/jobbeat # ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 78957
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 8596560
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 78957
virtual memory          (kbytes, -v) 10603440
file locks                      (-x) unlimited

We have a support contract so I did try and contact support but they were unavailable (in Australia), I will try and contact them again.

Thanks.

[MarkCulp]

Note that since you are running your server as a service the CWD of the process will be / ... so perhaps your core file(s) are being placed someplace else on your disk (if they are being generated). Check your /etc/sysctl.conf file (or /proc/sys/kernel/core_pattern) to see if a 'core_pattern' has been set on your system. Failing that you may want to run 'find . -name core\\* -print' to find all core files on your system.

[jeff_albion]

Customer Support is usually available 8 a.m - 6 p.m. (although this varies by country), Monday through Friday in your local timezone. See: http://www.sybase.com/contactus/support/ for details.