SQLAnywhere periodic crash on Windows 2008R2

[Former Member]

EDIT: solved!
The Problem was the ANSI version of the MySQL ODBC Driver. The crash happened when non-database-codepage characters should have been converted while selecting from the database's remote servers.

The crash does not happen when using the "MySQL ODBC 5.3 Unicode Driver" for the remote server. Now I have to change a few Powerbuilder applications but at least the database doesn't explode every few hours.

I don't know if this is a bug in the MySQL Connector or ASA's remote server though.

---

For the last week I get this crash every few hours:

VERSION=17.0.7.3399
FILENAME=C:\\ProgramData\\SQL Anywhere 17\\diagnostics\\SA17_20170606_123114_5208.crash_log
OS=Windows 2008R2 Build 7601 Service Pack 1
PROCESSOR=X86_64
EXEC_ARCH=X86_64
EXEC_PATH=D:\\APL\\SQL Anywhere 17\\Bin64\\dbsrv17.exe
MODULE_PATH=D:\\APL\\SQL Anywhere 17\\Bin64\\dbserv17.dll
EXCEPTION_PTR=0000000029EDF370
EXCEPTION_CODE=3221225477
EXCEPTION_FLAGS=0
EXCEPTION_RECORD=0000000000000000
EXCEPTION_ADDRESS=000000007712F23C
EXCEPTION_NumParameters=2
EXCEPTION_Param0=0000000000000000
EXCEPTION_Param1=000002789671CDC8
TRYING_TO_SAVE_MINI_DUMP C:\\ProgramData\\SQL Anywhere 17\\diagnostics\\SA17_20170606_123114_5208.dmp
DUMPLEVEL 0
SUBMIT_MINIDUMP=1

Stuff I tried:

dbvalid: no errors
upgrading to 17.0 SP0 PL20 Build 3399: did not help

Does anyone have a few tips to narrow down the problem?
The next step I will take is unloading and reloading the database but it's currently being used and I will have to wait till after-hours.

EDIT:
There is no logged assertion failure, the database just crashes. Today it did so twice within 5 minutes. Here is the Windows logs:

Log Name:      System
Source:        Service Control Manager
Date:          07.06.2017 10:32:40
Event ID:      7034
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mypc
Description:
The SQL Anywhere - ASA service terminated unexpectedly.  It has done this 8 time(s).

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          07.06.2017 10:32:40
Event ID:      4688
Task Category: Process Creation
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      mypc
Description:
A new process has been created.

Subject:
  Security ID:    SYSTEM
  Account Name:   $
  Account Domain:   A
  Logon ID:   0x3e7

Process Information:
  New Process ID:   0x1408
  New Process Name: X:\\SQL Anywhere 17\\Bin64\\dbsupport.exe
  Token Elevation Type: TokenElevationTypeDefault (1)
  Creator Process ID: 0x1dc
  Process Command Line: "X:\\SQL Anywhere 17\\Bin64\\dbsupport.exe" -crash "C:\\ProgramData\\SQL Anywhere 17\\diagnostics\\SA17_20170607_103239_476.crash_log" -crashpn "dbsrv17" -crashsn "ASA"

The exception code and address are always identical.

After checking the dump it looks like that is a problem with a remote server and the MySQL ODBC. I will try updating the MySQL ODBC drivers and see if that helps.

FAULTING_IP: 
ntdll!RtlFreeHeap+1a5
00000000`7712f23c 488b7b08        mov     rdi,qword ptr [rbx+8]

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000000007712f23c (ntdll!RtlFreeHeap+0x00000000000001a5)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 000001b9c99254c8
Attempt to read from address 000001b9c99254c8

PROCESS_NAME:  dbsrv17.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1:  0000000000000000
EXCEPTION_PARAMETER2:  000001b9c99254c8
READ_ADDRESS:  000001b9c99254c8

FOLLOWUP_IP: 
ntdll!RtlFreeHeap+1a5
00000000`7712f23c 488b7b08        mov     rdi,qword ptr [rbx+8]

MOD_LIST: <ANALYSIS/>
ADDITIONAL_DEBUG_TEXT:  Enable Pageheap/AutoVerifer
FAULTING_THREAD:  000000000000114c
DEFAULT_BUCKET_ID:  HEAP_CORRUPTION
PRIMARY_PROBLEM_CLASS:  HEAP_CORRUPTION
BUGCHECK_STR:  APPLICATION_FAULT_HEAP_CORRUPTION_INVALID_POINTER_READ_FILL_PATTERN_ffffffff
LAST_CONTROL_TRANSFER:  from 0000000076ee1a0a to 000000007712f23c

STACK_TEXT:  
00000000`2cb2f680 00000000`76ee1a0a : 00000000`0893dfa0 00000000`00000000 00000000`2bb7e530 00000000`00000226 : ntdll!RtlFreeHeap+0x1a5
00000000`2cb2f700 000007fe`f68e69d8 : 00000000`088ce764 000007fe`dd6dd7bf 00000000`00000000 00000000`00000000 : kernel32!HeapFree+0xa
00000000`2cb2f730 000007fe`f68f7b46 : 00000000`2bc0b1e0 00000000`00000032 00000000`2bb7e530 00000000`088f0450 : msvcr120!free+0x1c [f:\\dd\\vctools\\crt\\crtw32\\heap\\free.c @ 51]
00000000`2cb2f760 000007fe`dd6de055 : 00000000`2bc074a0 00000000`00000002 00000000`00000032 00000000`00000000 : msvcr120!setlocale+0xe2 [f:\\dd\\vctools\\crt\\crtw32\\misc\\setlocal.c @ 53]
00000000`2cb2f7c0 00000000`2bc074a0 : 00000000`00000002 00000000`00000032 00000000`00000000 00000000`00000032 : myodbc5a+0x1e055
00000000`2cb2f7c8 00000000`00000002 : 00000000`00000032 00000000`00000000 00000000`00000032 00000000`2c299c70 : 0x2bc074a0
00000000`2cb2f7d0 00000000`00000032 : 00000000`00000000 00000000`00000032 00000000`2c299c70 00000000`2bb74b38 : 0x2
00000000`2cb2f7d8 00000000`00000000 : 00000000`00000032 00000000`2c299c70 00000000`2bb74b38 00000000`00000000 : 0x32

SYMBOL_NAME:  heap_corruption!heap_corruption
FOLLOWUP_NAME:  MachineOwner
MODULE_NAME: heap_corruption
IMAGE_NAME:  heap_corruption
DEBUG_FLR_IMAGE_TIMESTAMP:  0
STACK_COMMAND:  ~23s; .ecxr ; kb
FAILURE_BUCKET_ID:  HEAP_CORRUPTION_c0000005_heap_corruption!heap_corruption

Is there a way to report the dump file (most *.dmp are 0 bytes but some look OK) manually? The server does not have internet connectivity. Should I open a support case?

[asa25]

EXCEPTION_CODE=3221225477 -> general access violation exception

When an error does not occur using dbvalid, I think that DB does not have a problem. You should rebuild DB if you are anxious.

[Vlad]

How can the corrupted file break down the DB server? Even any archiver shows you the message "corrupted file" instead of showing a nuclear mushroom.

[former_SQLA_member1694880]

The most obvious: check you hardware. Start from RAM :).

[Former Member]

Thanks for the idea, but my Server is running on a VMWare Server platform and I have no control over the host.
The Exception Address is always the same so I suspect a bug/incompatibilty.

Is there a way to manually send the crash reports/dump files? The server doesn't have internet connectivity.

[VolkerBarth]

Please have a look at the builtin dbsupport utility and its many options.

You might copy the contents from the server's diagnostics dir to a machine with internet access (and SA 17 installed, but without its "own" support data) and send support information from there or try to use dbsupport -cp on the server to specify a proxy host to submit reports. (Note: I have not used any of the two options myself, so I can't tell whether they will work...)

[Former Member]

Thanks, I found the files and analyzed a dump (see edit). I will update the MySQL ODBC and report back. Proxy wouldn't help. The server is isolated and can't even use a proxy to access the internet.

[VolkerBarth]

Then I'd recommend to try to copy the diag dir to another machine (which is possible, I hope!) with internet access and try to submit the information from there. As stated, don't know whether this will work but dbsupport -sa should send the contents of the diag dir. You might have to adapt the dbsupport.ini for the machine with the copied files.

If you need support, I guess you will have to open a support incident. This forum here is based on voluntary contributions and therefore is not meant for urgent cases.

[Former Member]

Thanks for the assist. Looks like the crashes go away when switching from MySQL ANSI ODBC to MySQL Unicode ODBC. A bit of a pain since the ASA DB collation is 1252LATIN1 and I now have to use lots of converting. It's a workaround but the crashes are gone.

[VolkerBarth]

I'd ask for the conversion issue as a separate question.