SA17 Periodic Crash - Windows Server 2019. Minidumps submitted

[former_SQLA_member1694953]

We are experiencing crashes of SQL Anywhere 17.0.10.5963 at two separate sites. In both cases, we are running the SQL Anywhere server as a windows service on Windows Server 2019 Standard.

The following error is logged in the Windows System Log:

The SQL Anywhere - 17 Server service terminated unexpectedly.  It has done this 2 time(s).

We have reviewed the -o (console) log and do not see any useful errors - there are no hints as to what may be triggering this, the log just stops processing events until we restart the SQL service.

At both sites, we have installed a .NET application that queries data (read-only) from the SQL Anywhere server every 30mins. The crash seems to align with the times the application is querying SQL Anywhere, but only happens on a rare occasion (eg. once every 1-2 days, whilst we are running this tool every 30 mins). The .NET application uses the Sap.Data.SQLAnywhere driver (17.0.7.3399 - https://www.nuget.org/packages/Sap.Data.SQLAnywhere/17.0.7.3399)

We have run dbvalid on the databases at both sites and no issues were reported.

I have several crash dumps for both sites (Crash Log and Windbg output below). In all cases the exception code is c0000005 (Access violation).

I have submitted the crash dumps using dbsupport with the following Submission ID's. As we don't have access to log a support ticket, we are currently unable to follow this up directly.

Submission ID: 6973838

Submission ID: 6973842

Any suggestions would be greatly appreciated

Crash Log:

VERSION=17.0.10.5963
FILENAME=C:\\ProgramData\\SQL Anywhere 17\\diagnostics\\SA17_20210617_141509_5412.crash_log
OS=Windows 2016 Build 17763 
PROCESSOR=X86_64
EXEC_ARCH=X86
EXEC_PATH=C:\\Sybase17\\SQL Anywhere 17\\Bin32\\dbsrv17.exe
MODULE_PATH=C:\\Sybase17\\SQL Anywhere 17\\Bin32\\dbserv17.dll
EXCEPTION_PTR=5DA9F384
EXCEPTION_CODE=3221225477
EXCEPTION_FLAGS=0
EXCEPTION_RECORD=00000000
EXCEPTION_ADDRESS=64ADAB8A
EXCEPTION_NumParameters=2
EXCEPTION_Param0=00000000
EXCEPTION_Param1=0000008C
TRYING_TO_SAVE_MINI_DUMP C:\\ProgramData\\SQL Anywhere 17\\diagnostics\\SA17_20210617_141509_5412.dmp
DUMPLEVEL 0
SUBMIT_MINIDUMP=1
SAVING_MINI_DUMP_COMPLETED
CRASH_LOG_COMPLETE
ATTEMPTING_TO_SUBMIT_MINIDUMP...

Windbg Output:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Unable to verify timestamp for dbsrv17.exe

KEY_VALUES_STRING: 1

    Key  : AV.Dereference
    Value: NullClassPtr

    Key  : AV.Fault
    Value: Read

    Key  : Analysis.CPU.mSec
    Value: 3405

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 60210

    Key  : Analysis.Init.CPU.mSec
    Value: 749

    Key  : Analysis.Init.Elapsed.mSec
    Value: 46827

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 89

    Key  : Timeline.Process.Start.DeltaSec
    Value: 14402

    Key  : WER.Process.Version
    Value: 17.0.10.5963


CONTEXT:  (.ecxr)
eax=1e7b5308 ebx=1fded808 ecx=00000000 edx=00000222 esi=00000001 edi=1fd01208
eip=6135ab8a esp=5dc1f4ec ebp=00000001 iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
dbserv17+0x40ab8a:
6135ab8a 39918c000000    cmp     dword ptr [ecx+8Ch],edx ds:002b:0000008c=????????
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 6135ab8a (dbserv17+0x0040ab8a)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 0000008c
Attempt to read from address 0000008c

PROCESS_NAME:  dbsrv17.exe

READ_ADDRESS:  0000008c

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  00000000

EXCEPTION_PARAMETER2:  0000008c

STACK_TEXT:  
5dc1f4e8 00000000     615e1076 00000222 1fd01208 dbserv17+0x40ab8a


SYMBOL_NAME:  dbserv17+40ab8a

MODULE_NAME: dbserv17

IMAGE_NAME:  dbserv17.dll

STACK_COMMAND:  ~6s ; .ecxr ; kb

FAILURE_BUCKET_ID:  NULL_CLASS_PTR_READ_c0000005_dbserv17.dll!Unknown

OSPLATFORM_TYPE:  x86

OSNAME:  Windows 10

IMAGE_VERSION:  17.0.10.5963

FAILURE_ID_HASH:  {536d8809-7503-dcb1-1c5d-81b485326593}

Followup:     MachineOwner

[Vlad]

Have you contacted SAP support via an incident? If this is a production down situation, you can even set the Very High priority.

[former_SQLA_member1694953]

Unfortunately I don't have access to SAP Support / Launchpad to raise an incident myself and it is taking some time to try to arrange this. Is there any other way? I have asked SAP about one-off paid incidents but was told unfortunately this is not possible.

I was hoping someone here might have access to look into the submission ID's, it seems this has been possible on some other threads. Aside from that any other recommendations or thoughts would be much appreciated.

[VolkerBarth]

If you do not have a SAP support contract, I assume you don't have access to newer SPs/EBFs, either. Otherwise, I would have suggested to try whether the newest version (17.0.10.6285 on Windows, AFAIK) both on server and client app solves this issue.

Of course you might use diagnostic tools like

• request level logging (RLL) or
• communication logging via dbsrv17 -z and the LOG connection parameter on the client

to look for more details.

[Breck_Carter]

This is ALL GUESSWORK based on the information you have posted.

Have you recently upgraded dbsrv17.exe from 17.0.7.3399 to 17.0.10.5963?

That Microsoft/Nuget package manager link you posted references build 17.0.7.3399 which is much older, and perhaps there is a conflict: https://www.nuget.org/packages/Sap.Data.SQLAnywhere/17.0.7.3399

...maybe someone else understands Nuget.

[Breck_Carter]

The CodeRush link from the Nuget link you posted shows these two entries (among others).

Note that the 17.0.7.3399 entry says "looking for v16..." and the 16.0.0.20433 entry says "looking for v17..."

...that looks suspicious to me 🙂

Sap.Data.SQLAnywhere by: CodeRush
17,526 total downloads
last updated 9/9/2017
Latest version: 17.0.7.3399
iAnywhere SQL Anywhere SyBase SAP .Net
SAP Data SQLAnywhere for .NET v3.5 and v4.5 (looking for v16 Install: iAnywhere.Data.SQLAnywhere.v4.5)

iAnywhere.Data.SQLAnywhere.v4.5 by: CodeRush
14,891 total downloads
last updated 12/3/2016
Latest version: 16.0.0.20433
iAnywhere SQL Anywhere SyBase SAP .Net 16
iAnywhere Data SQLAnywhere for .NET v3.5, v4.0 and v4.5 (Looking for version 17, install: Sap.Data.SQLAnywhere)

...never mind that 16.0.0.20433 never existed, and 16.0.0.2043 is much older than the most recent build for V16.

Question: Can you trust a website with docs this [cough] confusing?

[VolkerBarth]

Isn't that just a link to the according other version, as the namespace has changed between v16 and v17?

[Breck_Carter]

Doh! 🙂

[Breck_Carter]

Just guessing (again) but perhaps your hard drive is a more appropriate source than Nuget 🙂

Here's what my hard drive contains...

Folder: C:\\Program Files\\SQL Anywhere 17\\Assembly\\V4.5
  File: Sap.Data.SQLAnywhere.v4.5.dll

File description Sap.Data.SQLAnywhere for .NET 4.5
File version     17.0.10.62854
Product version  17.0.10.62854
Date modified    2021-03-19 11:58

I don't know where that "4" comes from. @@VERSION returns 17.0.10.6285 for me... maybe it's a check digit 🙂

[VolkerBarth]

AFAIK the added 5th digit tells the .Net major version, therefore a 4 here.

[former_SQLA_member1694953]

Thank you for picking up on this, this sounds like a solid next step

I've downloaded the latest SQLA17 developer version and it contains this DLL as at EBF 6089 (Production Version 17.0.10.60894). Whilst not the latest, its still 2.5 years newer than what we are currently using.

I'm guessing we wouldn't be able to access the latest version like you have there (17.0.10.62854) without a support contract?

[VolkerBarth]

I'm guessing we wouldn't be able to access the latest version like you have there (17.0.10.62854) without a support contract?

Yes, unfortunately that is SAP's support policy since summer 2014, see here.

FWIW, the read me files are published here, so at least you may have a look if your issue may have been addressed in a newer EBF. There are several entries dealing with crashes of the database server fixed since 17.0.10.5963.

(Of course I can't tell whether your issue would relate to a bug within SQL Anywhere at all or has a different origin...)

[Breck_Carter]

IMO if you are executing dbsrv17.exe 17.0.10.5963 then you should run the matching version of the dll, and that should have been automatically installed together with dbsrv17.exe, just in a different subfolder...

...so try looking there: C:\\Program Files\\SQL Anywhere 17\\Assembly\\V4.5

[VolkerBarth]

IMO if you are executing dbsrv17.exe 17.0.10.5963 then you should run the matching version of the dll

Do you think so? AFAIK, a database server should generally be compatible with clients of the same major version or even older clients... – although if the app runs on the database server box itself, I'd also recommend to use the same client version.

[Breck_Carter]

Someone didn't get the memo that clients don't cause server crashes 🙂

SQL Anywhere - ADO.Net Managed Provider
================(Build #4840  - Engineering Case #815956) 
Under some circumstances, a row constructed from a subquery 
could cause server crash when it was later assigned to another row. This has been fixed.

( when I say "someone" I don't mean "you", I mean engineering 🙂