Error "found wrong state to be transmitted" in TrexQueueServerAlert_<host>.trc

[stefan_sessner]

Hi,

since few days we are facing the following error messages in trace file /usr/sap/XXX/TRX99/<host>/trace/TrexQueueServerAlert_<host>.trc:

[140434960635648] 2014-06-02 14:27:46.893 e _Q_ses:uve_h Queue.cpp(04523) : Queue::preprocessMsg: found wrong state to be transmitted for 000000000010719878ES, Queue: ses:uve_hc_p_ardclnt101_bus1001006_1

[140435019376384] 2014-06-02 14:27:46.892 e _Q_ses:uve_h Queue.cpp(04523) : Queue::preprocessMsg: found wrong state to be transmitted for 000000000011110089DE, Queue: ses:uve_hc_p_ardclnt101_bus1001006_1

[140435036456704] 2014-06-02 14:27:46.892 e _Q_ses:uve_h Queue.cpp(04523) : Queue::preprocessMsg: found wrong state to be transmitted for 000000000011110089EN, Queue: ses:uve_hc_p_ardclnt101_bus1001006_1

[140434977421056] 2014-06-02 14:27:46.892 e _Q_ses:uve_h Queue.cpp(04523) : Queue::preprocessMsg: found wrong state to be transmitted for 000000000014435640ES, Queue: ses:uve_hc_p_ardclnt101_bus1001006_1

These entries are written every second in this trace file. Has anyone a hint, an explanation or a solution for this strange behavior?

What does this message mean? I cannot find anything in SAP Support Portal or here in SCN. Thanks in advance.

Kind regards, Stefan

[former_member217429]

Hi Stefan,

the mentioned error messages are usually recorded when the queue is already inconsistent or corrupted. However these are just the "follow-up" events in the trace and the root cause of the problem is usually to be found on the top of the trace. Is the trace already very big? I would like to recommend to suspend the queue (TREXAdminTool -> Queue -> Admin -> Select Queue ses:uve_hc_p_ardclnt101_bus1001006_1 -> suspend . Most likely the index has to be recreated from tc. ses_admin. However before doing this it's necessary to identify the root cause of the issue. It could be the bug in TREX, hardware or 3rd party software (antivirus , backup - s. note 808768) on the server. Are there any other error messages in the QueueServer or IndexServer traces ?

Best regards,
Mikhail

[stefan_sessner]

Thx. So we checked note 808768 and deactivated access of backup tool (Networker) to index and queue directories. There aren't any other error messages in the QueueServer or IndexServer traces. Meanwhile these traces are huge. How can we find out the root cause in these traces? Will stopping TREX, clearing all trace files and starting TREX again, bring out a root cause trace entry?

[former_member217429]

Hi Stefan,

no, unfortunately the restart , cleanup of the traces doesn't help to find out the root cause of the issue. Is the TREXIndexServerAlert trace huge too? Usually this should be much less than the TrexQueueServer trace (also if the queues are corrupted). Is it possible for you to upload this trace into SAP Box Attachment

container. Probably I can find something there. As I understand this is the installation on the Linux. Correct? Are the any error messages in the /var/log/messages log related to TREX or issues on the File System? What's about the TREX version and the OS version you use (f.e. is it SLES 11 SP 2/3 and TREX 710 REV 48)?

Best regards,
Mikhail

[stefan_sessner]

No, the TREXIndexServerAlert trace is only about 100 KB. I uploaded this trace file already. There are some "out of disk space" errors, but this I guess is the result of growing of TrexQueueServerAlert trace up to 80 GB meanwhile. And yes, Linux SLES 11 SP 3 and TREX 710, but already Rev. 49.

[former_member217429]

Hi Stefan,

yes, it could be possible that the "out of disk space"  is the result of the growing the QueueServer log. This error appears in the IndexServer trace on 21.05 first:2014-05-21 07:01:19.449 e SERVER_TRACE DeltaIndexManager.cpp(01684) : prepare failed for 162d4e489a2c611fe1000000c3d93438p41101dewith rc=1507

However related to the other one index. I don't see anything in the IndexServer trace related to the

ses:uve_hc_p_ardclnt101_bus1001006_1 index.
We have had an issue with the incomartibility between the SLES 11 SP 2 and the TREX, but this should be fixed with the REV 49. Do you know when the SP 3 was installed on this TREX server?
What's the current size of the TrexQueueServerAlert trace? Can you try to search for uve_hc_p_ardclnt101_bus1001006_1 and check one the 1st one error related to this index.
As for now, it looks like that the index/queue are corrupted . I would like to recommend to delete this index from SES_ADMIN - > update the TREX to the REV 58 -> recreate index.

Best regards,
Mikhail