SA service disappears without a trace?

[justin_willey]

Does anyone have any ideas on where to start looking when a database engine service simply disappears? No assertion, no error in the console log, nothing in the Windows event logs.

Server re-starts without complaint.

v10.0.1.4310

[VolkerBarth]

Anything in the translated transaction log?

[justin_willey]

Hi Volker. I've translated it now, but there's nothing except a gap in time between normal activity and then a check point when the server came back up three minutes later:

--INSERT-1582-02223301401
INSERT INTO pears.RecentObjects(StaffID,ObjectType,ObjectID,LastViewed)
VALUES ('MG0OYXCG140620110000','T','X459479','2012-09-20 11:21:11.967')
go
--COMMIT-1582-02223301446
COMMIT WORK
go
--CHECKPOINT-0000-02223301471-2012-09-20 11:24
--CONNECT-1287-02223301497-pears-2012-09-20 11:24
--BEGIN TRANSACTION-1287-02223301510

The Recent objects record is inserted automatically and the timestamp on it (11:21:11) is just before the time the server died. The checkpoint at 11:24 matches the restart.

[VolkerBarth]

Have you used "DBTRAN -a -d" or other switches to "get most out of the log" for the according log part - say, in order to notice any uncommitted operations?

Besides that, I can't tell whether the contents gives any clues:(

[justin_willey]

good point re the -a , re-translating now.

[justin_willey]

This is the version with the -a -d switches

--INSERT INTO pears.RecentObjects(StaffID,ObjectType,ObjectID,LastViewed)
--VALUES ('MG16KIPA250820090000','P','MG14VXKC300420070039','2012-09-20 11:21:11.889')
--COMMIT-1278-02223301310
--COMMIT WORK
--UPDATE-2363-02223301311
--UPDATE pears.TempShift
--   SET PlacementID='MG17ADJF120620120047',
--       TempTimeSheetID='MG0VJBLH2009201200B7',
--       State='W',
--       BreakMinutes=40
-- WHERE TempShiftID='MG0T4YDD310820120006'
--CONNECT-2363-02223301447-recovery-2012-09-20 11:24
----ROLLBACK-2363-02223301463
--ROLLBACK WORK
--ROLLBACK-2363-02223301464
--ROLLBACK WORK
--CHECKPOINT-0000-02223301471-2012-09-20 11:24
--CONNECT-1287-02223301497-pears-2012-09-20 11:24

but no smoking gun that I can see...

[VolkerBarth]

A case for the great iAnywhere experts, obviously... - I hope they come around here:)

[VolkerBarth]

Sorry, I couldn't resist to note that this isn't the supposed meaning of the slogan "SQL Anywhere - The Invisible Database"...

[jeff_albion · Accepted solution]

When the SQL Anywhere database server unexpectedly crashes, it will normally catch this event itself and automatically create a mini-dump file and supply this information to dbsupport. Depending upon your submission settings, dbsupport will either auto-submit this information automatically to SAP/Sybase or will hold on to the information until you submit it manually.

Use dbsupport -lc to list all unsubmitted crashes on the system. Use dbsupport -sa to submit any outstanding crashes to SAP/Sybase. Once the information has been submitted, if you post the submission ID here, I can take a quick look at the submission for you to see if it's a known issue.

If your server is still crashing and no information is being passed to dbsupport, there are other ways to manually collect this diagnostic information from the server (via procdump and psexec, if required). If you need to go this route though, we would recommend that you also open a technical support case with us so that we can work with you on collecting this output and have it analyzed.

A reminder that SQL Anywhere 10.0.x has been end-of-life'd, so there could also be an outstanding technical issue with the product that will not be fixed by a software patch - we will attempt to provide "limited technical assistance" for the product via technical support instead (suggest workarounds, etc.).

---

"Crashes" can frequently be related to database corruption (although this will typically be reported as an "assertion" in the database console log - I noted that you've already checked this: dbsrv10 -o console.txt). Have you validated your database to ensure it is not corrupted? Particularly the pears.TempShift and pears.RecentObjects tables? If you re-run the last INSERT and UPDATE logged in the transaction log, does it result in the same crash?

---

Update by Justin Willey - support found the problem in a query evaluation / optimizer routine. The routine was rewritten for v11 so shouldn't be present in v11 or higher versions. See comments below.

[justin_willey · Accepted solution]

Mnay thanks for the detailed reply Jeff - I'm seeing what we can find.

On the point about corruption we have a full validation scheduled for Saturday. Also I can try those specific statements again as you suggest.

[justin_willey · Accepted solution]

Hi - just had a recurrence - have tried submitting but getting:

Error submitting report: No reply received from web server. Error sending to www.ianywhere.com:80.

Is it possible to send you the files?

Many thanks

[jeff_albion · Accepted solution]

I believe we have a current outage in the submission server - I'm not sure when this is to be resolved.

Do you have a support plan and can you open a support ticket for this issue?

[justin_willey · Accepted solution]

@Jeff, I've submitted the mini core files etc for event SA10_20121017_101730_3440 (this is another incident yesterday with the same symptoms) under case #11759414 - many thanks

[justin_willey · Accepted solution]

The (full) validation completed without any problems.

[justin_willey · Accepted solution]

Have heard back from support. The crash seems to relate to a portion of code in the query evaluation / optimization process. The routine in question has been re-written in v11 so the problem shouldn't be present in that or later versions.