Replication suddenly stopped ?

[thelmacottage]

One of our sites has stopped replicating - the messages are being sent from the remote site to the consolidated but the log shows the following message ?

W. 2018-07-11 12:06:44. This message does not belong to me.
                        Log offset sent to user 'STORE18' is '0639618234222', but the message header indicates user 'STORE18' has received log offset '0639815876528'.
I. 2018-07-11 12:06:44. Received message from "STORE18" (1040-0139647190868-0139647190868-0)

It was okay up to 8am this morning - not quite sure what to do !

[VolkerBarth]

Do further attempts from that remote succeed?

Has the remote been restored from an older backup, and it had already replicated in the meantime (i.e. between doing the backup and restoring the database)?

[thelmacottage]

Remote has not been restored it has been running fine until 8 this morning when it started giving this message in the consolidated logs.

I re-started the replication service at both ends and now it seems to RECEIVE to messages but not apply them ?

I have now quite a lot of messages sitting in the CONSOL folder waiting to be processed (just from that site) - they don't seems to be doing anything.

This is quite worrying !

I. 2018-07-11 13:39:40. Received message from "STORE01" (4-019917795343-019917797052-0)
I. 2018-07-11 13:39:40. Applying message from "STORE01" (4-019917795343-019917797052-0)
I. 2018-07-11 13:39:40. Received message from "STORE08" (1-09782927316-09782930392-0)
I. 2018-07-11 13:39:40. Applying message from "STORE08" (1-09782927316-09782930392-0)
I. 2018-07-11 13:39:40. Received message from "STORE11" (14-07876584636-07876599089-0)
I. 2018-07-11 13:39:40. Applying message from "STORE11" (14-07876584636-07876599089-0)
I. 2018-07-11 13:39:40. Received message from "STORE14" (18-015749822952-015750111039-0)
I. 2018-07-11 13:39:40. Applying message from "STORE14" (18-015749822952-015750111039-0)
I. 2018-07-11 13:39:40. Received message from "STORE17" (18-058075812157-058075831249-0)
I. 2018-07-11 13:39:40. Applying message from "STORE17" (18-058075812157-058075831249-0)
I. 2018-07-11 13:39:41. Received message from "STORE18" (1040-0139689998212-0139690186658-0)
I. 2018-07-11 13:39:41. Received message from "STORE18" (1040-0139690186658-0139690186658-0)
I. 2018-07-11 13:39:41. Received message from "STORE18" (1040-0139690186658-0139690186658-1)
I. 2018-07-11 13:39:41. Received message from "STORE18" (1040-0139690186658-0139690186658-2)
I. 2018-07-11 13:39:41. Received message from "STORE18" (1040-0139690186658-0139690262799-3)
I. 2018-07-11 13:39:41. Received message from "STORE19" (29-0120629734746-0120629736110-0)
I. 2018-07-11 13:39:41. Applying message from "STORE19" (29-0120629734746-0120629736110-0)
I. 2018-07-11 13:39:41. Received message from "STORE19" (29-0120629736110-0120629736110-0)
I. 2018-07-11 13:39:41. Received message from "STORE19" (29-0120629736110-0120629774155-1)
I. 2018-07-11 13:39:41. Applying message from "STORE19" (29-0120629736110-0120629736110-0)
I. 2018-07-11 13:39:41. Applying message from "STORE19" (29-0120629736110-0120629774155-1)

[VolkerBarth]

What I usually do in these cases is try to compare the contents of SYSREMOTEUSER on both databases for the according row (i.e. within the cons, for the according remote). Do you have access to the contents of SYSREMOTEUSER on the remote database, say, via remote access?

That way you can usually find out what both parties know about each other and find possible problens.

The message in your question basically means the remote claims to have received newer (*) messages from the cons than the cons knows to have sent to that remote.

(*) "Newer" meaning "with a higher log offset", not necessarily with an newer timestamp.

[thelmacottage]

Finally managed to get the remote store to send all it's messages again (the consol found a 'missing message) - however it took several hours to copy the 4000 messages that was generated to the consolidated !! - I did get lot's of 'corrupt message' warnings on the consolidated so had to do this several times. It always seems to be the same REMOTE SITE that causes the problem.

[thelmacottage]

This site is playing up again. This time I have 38,000 messages to be copied over ! - Not sure what's wrong with this site.

Is there anyway to avoid copying over this amount of messages ?