dbsrv16 crash while unloading database

[Former Member]

Problem with customer database: dbsrv16 keeps crashing while unloading a database. it crashes always at the same table with failure: "Assertierung fehlgeschlagen" seems to fail always at unloading the same table.

What can cause such a problem? I have inconsistencies in another databases primekey index too, used by the same software.

Appending the crash details here:

Server SQL Anywhere 16 16.0.0.2322:

Command dbunload -v -c "UID=sirius;PWD=***;Server=sa16jw;DBF=/home/jw/Kunden/PerformanceTest/Database/PerformanceTest.db;LINKS=TCPIP()" -r "/home/jw/Kunden/PerformanceTest/reload.sql" -ii "/home/jw/Kunden/PerformanceTest/unload"

ProblemType: Crash
Architecture: amd64
CurrentDesktop: i3
Date: Wed Feb  8 17:27:28 2017
DistroRelease: Ubuntu 16.04
ExecutablePath: /opt/sybase/sqlanywhere16/bin64/dbsrv16
ExecutableTimestamp: 1478271536
ProcCmdline: dbsrv16 -ui -gd all -cl 1g -c 4g -n sa16jw
ProcCwd: /var/crash
ProcEnviron:
 LANGUAGE=de_DE:fy:en
 LC_TIME=de_DE.UTF-8
 LD_LIBRARY_PATH=<set>
 LC_MONETARY=de_DE.UTF-8
 TERM=xterm-256color
 PATH=(custom, user)
 LC_ADDRESS=de_DE.UTF-8
 XDG_RUNTIME_DIR=<set>
 LC_TELEPHONE=de_DE.UTF-8
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
 LC_NAME=de_DE.UTF-8
 LC_MEASUREMENT=de_DE.UTF-8
 LC_IDENTIFICATION=de_DE.UTF-8
 LC_NUMERIC=de_DE.UTF-8
 LC_PAPER=de_DE.UTF-8
ProcMaps:
 00400000-00403000 r-xp 00000000 fc:00 8436                               /opt/sybase/sqlanywhere16/bin64/dbsrv16
 00602000-00603000 rw-p 00002000 fc:00 8436                               /opt/sybase/sqlanywhere16/bin64/dbsrv16
 024be000-0264a000 rw-p 00000000 00:00 0                                  [heap]
 7fb168000000-7fb168021000 rw-p 00000000 00:00 0 
 7fb168021000-7fb16c000000 ---p 00000000 00:00 0 
 7fb170000000-7fb170021000 rw-p 00000000 00:00 0 
 7fb170021000-7fb174000000 ---p 00000000 00:00 0 
 7fb174000000-7fb174021000 rw-p 00000000 00:00 0 
 7fb174021000-7fb178000000 ---p 00000000 00:00 0 
 7fb178000000-7fb178021000 rw-p 00000000 00:00 0 
 7fb178021000-7fb17c000000 ---p 00000000 00:00 0 
 7fb17c000000-7fb17c021000 rw-p 00000000 00:00 0 
 7fb17c021000-7fb180000000 ---p 00000000 00:00 0 
 7fb180000000-7fb180021000 rw-p 00000000 00:00 0 
 7fb180021000-7fb184000000 ---p 00000000 00:00 0 
 7fb1852fc000-7fb1852fe000 r-xp 00000000 fc:00 5027                       /opt/sybase/sqlanywhere16/lib64/libdbscript16_r.so.1
 7fb1852fe000-7fb1854fd000 ---p 00002000 fc:00 5027                       /opt/sybase/sqlanywhere16/lib64/libdbscript16_r.so.1
 7fb1854fd000-7fb1855f9000 rw-p 00001000 fc:00 5027                       /opt/sybase/sqlanywhere16/lib64/libdbscript16_r.so.1
 7fb1855fa000-7fb1855fa000 ---p 00000000 00:00 0 
 7fb1855fa000-7fb1857fb000 rw-p 00000000 00:00 0 
 7fb1857fb000-7fb1857fc000 ---p 00000000 00:00 0 
 7fb1857fc000-7fb185ffc000 rw-p 00000000 00:00 0 
 7fb185ffc000-7fb185ffd000 ---p 00000000 00:00 0 
 7fb185ffd000-7fb1867fd000 rw-p 00000000 00:00 0 
 7fb1867fd000-7fb1867fe000 ---p 00000000 00:00 0 
 7fb1867fe000-7fb186ffe000 rw-p 00000000 00:00 0 
 7fb186ffe000-7fb186fff000 ---p 00000000 00:00 0 
 7fb186fff000-7fb1877ff000 rw-p 00000000 00:00 0 
 7fb1877ff000-7fb187800000 ---p 00000000 00:00 0 
 7fb187800000-7fb188000000 rw-p 00000000 00:00 0 
 7fb188000000-7fb188021000 rw-p 00000000 00:00 0 
 7fb188021000-7fb18c000000 ---p 00000000 00:00 0 
 7fb18c000000-7fb18c021000 rw-p 00000000 00:00 0 
 7fb18c021000-7fb190000000 ---p 00000000 00:00 0 
 7fb190000000-7fb190021000 rw-p 00000000 00:00 0 
 7fb190021000-7fb194000000 ---p 00000000 00:00 0 
 7fb1941e0000-7fb194434000 r-xp 00000000 fc:00 7921                       /opt/sybase/sqlanywhere16/lib64/libdbldap16_r.so.1
 7fb194434000-7fb194633000 ---p 00254000 fc:00 7921                       /opt/sybase/sqlanywhere16/lib64/libdbldap16_r.so.1
 7fb194633000-7fb194663000 rw-p 00253000 fc:00 7921                       /opt/sybase/sqlanywhere16/lib64/libdbldap16_r.so.1
 7fb194663000-7fb194669000 rw-p 00000000 00:00 0 
 7fb194669000-7fb19466a000 ---p 00000000 00:00 0 
 7fb19466a000-7fb194e6a000 rw-p 00000000 00:00 0 
 7fb194e6a000-7fb194e6b000 ---p 00000000 00:00 0 
 7fb194e6b000-7fb19566b000 rw-p 00000000 00:00 0 
 7fb19566b000-7fb19566c000 ---p 00000000 00:00 0 
 7fb19566c000-7fb195e6c000 rw-p 00000000 00:00 0 
 7fb195e6d000-7fb195e6d000 ---p 00000000 00:00 0 
 7fb195e6d000-7fb195f6e000 rw-p 00000000 00:00 0 
 7fb195f6e000-7fb195f6f000 ---p 00000000 00:00 0 
...
ProcStatus:
 Name:  dbsrv16
 State: D (disk sleep)
 Tgid:  21072
 Ngid:  0
 Pid:   21072
 PPid:  1
 TracerPid: 0
 Uid:   1000    1000    1000    1000
 Gid:   1000    1000    1000    1000
 FDSize:    64
 Groups:    4 20 24 46 116 118 124 125 1000 
 NStgid:    21072
 NSpid: 21072
 NSpgid:    21071
 NSsid: 20455
 VmPeak:    24424236 kB
 VmSize:    24363888 kB
 VmLck:        0 kB
 VmPin:        0 kB
 VmHWM:   469828 kB
 VmRSS:   469828 kB
 VmData:    23879424 kB
 VmStk:    89012 kB
 VmExe:       12 kB
 VmLib:    52528 kB
 VmPTE:     1732 kB
 VmPMD:       28 kB
 VmSwap:           0 kB
 HugetlbPages:         0 kB
 Threads:   33
 SigQ:  2/96107
 SigPnd:    0000000000000000
 ShdPnd:    0000000000000000
 SigBlk:    0000000000004007
 SigIgn:    0000000000001000
 SigCgt:    0000000180000000
 CapInh:    0000000000000000
 CapPrm:    0000000000000000
 CapEff:    0000000000000000
 CapBnd:    0000003fffffffff
 CapAmb:    0000000000000000
 Seccomp:   0
 Cpus_allowed:  00ff
 Cpus_allowed_list: 0-7
 Mems_allowed:  00000000,00000001
 Mems_allowed_list: 0
 voluntary_ctxt_switches:   208
 nonvoluntary_ctxt_switches:    39
Signal: 6
Uname: Linux 4.4.0-59-generic x86_64

[VolkerBarth]

What does a full validation of the database reveal, say via dbvalid?

[Former Member]

both dbvalid and "VALIDATE DATABASE" in ISQL lets the server crash.

FEHLER "Assertierung fehlgeschlagen" ... Checksum error on page ....

[Former Member]

to reproduce: example db for testing is available here: http://k00.fr/27xde (about 600 MB, including sybase crash log and dump)

[Breck_Carter]

There used to be tech note titled "I've Got An Assertion! What Should I Do?" but I can no longer find it.

You may have a corrupted disk, or RAM errors, or a combination of both.

In an ideal world, you will restore the database to a different computer, from a recent backup.

In a less-than-ideal world, you may not have a usable backup, so...

If you suspect RAM errors, further attempts to use the database may result in permanent disk errors as corrupt RAM values are written to the disk. It is suggested you stop updating the database until the data can be rescued.

If you already know which table is having problems, you can uses dbunload with -t to unload all the other tables.

If the primary key is having problems, try dropping the primary key before unloading that table.

Do not write the unloaded data to the same suspect disk; consider using an external USB 3 drive.

Run an extended RAM check on the suspect computer, or just replace the RAM (especially if you have recently installed suspect chips).

Same thing with the disk drive, to make sure any corruption is transient; i.e., bad stored values rather failing bits.

Do that before trying to rebuild the database from the unloaded data.

[Former Member]

thanks for your comprehensive answer. the hardware was checked, but with no results (no failures show out on RAM and HD).

[VolkerBarth]

Here's a link from the SQL Anywhere Wiki to the topic mentioned by Breck:

Handling an Assertion Failure

I guess that's the most official document on that topic currently...

[Former Member]

Thanks for the link. We repaired the database by unloading it (without the table containing the defect), reloading to new database and adding the table again.