Sybase Event Stream Processor error messages generally follow a standard format. However, ESP uses a number of open source components. The format of the error message depends on which component throws the error. This blog post attempts to shed a little light on some of the various ESP server error messages. A future blog will touch on errors thrown by other components.
The errors most relevant to the project's runtime state will show up in ESP's "esp_server.log" file. Finding this log file can be tricky but the documentation offers some good guidance (Project Logging). The second most important log file related to the project is the "stdstreams.log" file which contains version and licensing information.
esp_server.log
Here is an example from my "esp_server.log":
2013-04-05 16:43:59.725 | 110 | container | [SP-3-148001] (23.418) sp(24460) ConnectionReader(StockQuoteInStream) connection reset error: 2013-04-05 16:43:59 reset(): could not read file '/software/ESP51/ESP-5_1/sampledata/STOCK_QUOTE.csv'
Breaking down the format:
YYYY-MM-DD HH:MM:SS.SSS | ZZZ | container | [SP-XX-MMMNNN] (seconds) sp(pid) ModuleName(Window_or_Stream) error message text
YYYY-MM-DD HH:MM:SS.SSS - The UTC timestamp for the message.
ZZZ - The ID of the lwp/thread that printed the message (this would be helpful if we were examining a core file and needed to match the message up with a thread).
container - The type of node (container is the only value I have ever seen).
XX - is the logging level:
| Name | Level | Description |
|---|
| LOG_EMERG | 0 | system is unusable |
| LOG_ALERT | 1 | action must be taken immediately |
| LOG_CRIT | 2 | critical conditions |
| LOG_ERR | 3 | error conditions |
| LOG_WARNING | 4 | warning conditions |
| LOG_NOTICE | 5 | normal but significant condition |
| LOG_INFO | 6 | informational |
| LOG_DEBUG | 7 | debug-level messages |
MMM - The module in which the message is logged (this number is mostly relevant to SAP technical support & engineering).
NNN - The message number for a given module. Each module could have multiple error messages and by combining the module number "MMM" and the error number "NNN", you get a unique error number.
seconds - The number of seconds after the Event Stream Platform startup that this message was generated (useful for troubleshooting).
pid - The process id for the Streaming Processor process.
ModuleName - The module or section of code that raised the error. Quite possibly it could be the name of the source file where the error was raised.
Window_or_Stream - If the error was tied to a particular stream or window, the name will be displayed.
Detailed analysis of an error message:
Now, let's go back and look at the example I provided:
2013-04-05 16:43:59.725 | 110 | container | [SP-3-148001] (23.418) sp(24460) ConnectionReader(StockQuoteInStream) connection reset error: 2013-04-05 16:43:59 reset(): could not read file '/software/ESP51/ESP-5_1/sampledata/STOCK_QUOTE.csv'
We know the date and time the error was logged (2013-04-05 16:43:59.725) and the thread ID (110). Since it was logged at level 3, it was an error so we should pay attention to it. I normally run my projects with the logging level set to 4 and I pay close attention to messages logged at level 3 or lower.
We see the unique six digit error number is "148001" for this particular error. ESP has an Error Message Guide where you can look up most of the error messages. Some of the newer messages have not been added to the guide but we hope to have them added in the future (and possibly added to a WIKI). The next thing I would do to understand this error is look up the number in the Error Message Guide:
In this case it suggests the resolution forerror number 148001 as:
- A connection failed to reset. Probably some parameters are specified wrong. Check and correct the model.
Not exactly the most helpful suggestion. What line of code should I look at in my project? What parameters need to be corrected? If we look at the rest of the message, it makes more sense.
The "ConnectionReader" module that raised the error might be a good place for the technical support engineer to start looking if they needed to review the code to gain a better understanding of the error. But for everyone else, this bit might not be that meaningful.
The name of the stream or window "StockQuoteInStream" is a good clue. We can search through the project's CCL code to see where this stream is referenced.
The last bit of the error is the most telling:
could not read file '/software/ESP51/ESP-5_1/sampledata/STOCK_QUOTE.csv'
This tells me that the stream "StockQuoteInStream" likely had a CSV Input Adapter (because it was trying to read the file) attached to it and that it was unable to open the file. I could now check to see if the file exists, if I have permission to read it or see if there was perhaps a typo in my project's CCL.
stdstreams.log
As I mentioned, the second most important log related to the project's runtime state is the "stdstreams.log". This log file contains the product's actual version string which is always important to know when contacting technical support. For example:
Sybase Event Stream Processor Engine 5.1.0/20130417.1/SP02 PL00/linux/x86_64/64-bit/OPT/Wed Apr 17 22:53:14 PDT 2013
It also contains messages related to licensing issues. The "esp_server.log" always has a message about a license check:
2013-08-14 19:46:33.974 | 18122 | container | [SP-3-100005] (0.000) sp(18122) ------ License check started ------
2013-08-14 19:46:34.004 | 18122 | container | [SP-3-100005] (0.030) sp(18122) ------ License check succeeded ------
This message is a bit misleading. It might give you the impression that all is well with your license but all it really tells us is that it ran a license check.
If you look at the "stdstreams.log", all might not be so well:
2013/08/14 19:46:34 Warning (131254): Checked out graced license for 1 ESP_CORE (2013.0417) will expire Fri 23 Aug 2013 12:00:00 AM UTC.
2013/08/14 19:46:34 Error (131302): Failed to obtain license(s) for ESP_CORE feature from license file(s) or server(s).
2013/08/14 19:46:34 Error (131073): Cannot find license file.
2013/08/14 19:46:34 Error (0): The license files (or license server system network addresses) attempted are listed below.
2013/08/14 19:46:34 Error (0): License feature name: ESP_CORE
2013/08/14 19:46:34 Error (0): License filename: /work/nstack/ESP51_SP2/SYSAM-2_0/licenses
2013/08/14 19:46:34 Error (0): License search path: /work/nstack/ESP51_SP2/SYSAM-2_0/licenses
2013/08/14 19:46:34 Error (0): FLEXnet Licensing error:-1,359. System Error: 2 "No such file or directory"
2013/08/14 19:46:34 Error (0): For further information, refer to the Sybase Software Asset Management website at http://www.sybase.com/sysam
Immediately I can spot a few problems:
- It checked out a "graced license" which means it is a temporary license (normally good for 30 days).
- It expires in a few days. I need to get a new license or fix the problem with the license I have or else I won't be able to run any projects.
- Error 131073 "Cannot find license file" is the underlying reason for it being unable to checkout a license.
- It tells me exactly where it was looking for the licenses.
So if your project ever fails to start, be sure to look at both the "esp_server.log" and the "stdstreams.log".
Hopefully this blog has helped take out some of the mysteries of ESP server error logs and given you a start on where to look when you want to troubleshoot an ESP problem.
