How To Diagnose Crashed Service

[former_SQLA_member1694868]

One of our clients is using version 12, build 2601.

We run the database as a Windows service on their 2008 Server.

Starting Jan 03 the service crashes with an event in the Windows event log "Service has terminated unexpectedly. The first time I restarted it, it stayed up for 2 weeks. After the next crash about 5 days, and now it has happened twice this week.

Beyond the Windows event log, which does not show any correlated events, what is the best way to diagnose what is happening just before the crash?

We have multiple clients with this setup and only one is having this problem, so I strongly suspect the environment, but I have to prove it. Any thoughts on what in the environment would cause this would also be helpful.

[Breck_Carter]

Did the server create a dump file when it crashed?

[former_SQLA_member1694868]

If you are asking about SA generating a dump file, I do not know how to check that. If you are asking about the server itself (like a blue-screen dump file), it did not crash. The application reports a communication error, I log in to the server and the service is stopped, and the event log has an entry "XYZ service has terminated unexpectedly." But the rest of the server seems fine and no other problems are reported.

[Former Member · Accepted solution]

If the server did catch the crash and generate a dump file, you should be able to see these by executing

dbsupport -lc
dbsupport -ls

If the -lc option has anything listed, you can try to submit these using

dbsupport -sa

The dumps themselves need to be analyzed by an engineer and the easiest way to get this to happen is to open a support case, and provide your case contact with the submission IDs from running dbsupport -ls.

---

That being said, the server dumps are often minimal and won't always give us the picture we need to determine what the problem is. We may ask for more info.

What you may want to check for yourself is server output messages before the crash occurs; these are output to a file using "-o server_log.txt" in the command line arguments if you are not already doing so.

If you find an assertion in the output log, you will likely want to try rebuilding a copy of the database, and seeing if the crash no longer occurs.

It could be that there are also other errors that occur in the output log, these will usually be prefixed with a E. on the line.

If you do have any trouble finding the cause of the crash, I would recommended opening a support case for investigation.

[VolkerBarth · Accepted solution]

I fully agree that dbsrvX -o (possibly with its variants -oX) should be using on every production server to help diagnosis.

[Breck_Carter · Accepted solution]

Further to what Tyson and Volker said: -o filespec1.txt -oe filespec2.txt -os 1M is a good combination for production. The -os 1M keeps the file size manageable for browsing with Wordpad etcetera. For a service the filespecs probably have to have the full path.

[former_SQLA_member1694868 · Accepted solution]

Excellent. Thanks everyone. Using the -o logging options I was able to catch the last message before the crash, which detailed the user and connection information. Version 9 ODBC connected to the 12 Database with the wrong version of our program as well.

[Former Member]

Will the version 9 odbc crash the asa12 database?

[former_SQLA_member1694868]

I do not know that this issue was the direct cause. I was listing all the things I was able to find as a result of the logging, which prompted me to update our app, the ODBC driver, and the server with the latest EBFs. The real lesson was not to trust a client saying everything was up to date, but instead log the connections and check for yourself.

[VolkerBarth]

In general, you can connect with older clients to newer servers.

For all current SA versions, clients from v6 and above should be able to connect (though they might not be able to use all newer client facilities like Unicode, Kerberos, connection pooling and the like). So a v9 client should not crash the database...

Have a look at the comments on this DCX page.