Clusters getting disappear and causing Caching issue.

[former_member664881]

Hi All,

We are using Hybris 6.5 version and TCP protocol for clustering and have 2 App nodes(App node1 and App node2) and 1 Admin node. Many a times it is seen that HAC shows no clusters in Admin node but 2 or 3 clusters on App nodes. If we refresh at the individual node, then sometime Clusters are shown and next second if we do refresh then sometimes clusters get disappear. We are facing issue that when some information (like Price Row update, Image update or Product description update) it is updated only on Admin node but not in App Node 1 or App Node 2. But when we clear cache on individual node, then information is updated on all the nodes. This is causing is a lot of problem. Any solution is well.

Below are some of the Log messages noticed at different times:-

1,WARN [Incoming-30930,hybris-broadcast,hybrisnode-1] [NAKACK] JGRP000011: hybrisnode-1: dropped message 1762420 from non-member hybrisnode-0 (view=[hybrisnode-2|3916] (2) [hybrisnode-2, hybrisnode-1])

2, WARN [Incoming-30935,hybris-broadcast,hybrisnode-1] [NAKACK] JGRP000011: hybrisnode-1: dropped message 1762819 from non-member hybrisnode-0 (view=[hybrisnode-2|3916] (2) [hybrisnode-2, hybrisnode-1]) (received 400 identical messages from hybrisnode-0 in the last 74086 ms)

3, WARN [INT-1,hybris-broadcast,hybrisnode-0] [FD] hybrisnode-0: I was suspected by hybrisnode-1; ignoring the SUSPECT message and sending back a HEARTBEAT_ACK INFO | jvm 1 | main | 2018/02/19 16:48:24.308 | INFO [RuleEngine-module-swapping-0] [DefaultRuleEngineKieModuleSwapper] Drools Engine Service initialization for 'promotions-module' module in tenant 'master' finished. ReleaseId of the new Kie Module: 'hybris-rules:promotions:1.0.0.173' INFO | jvm 1 | main | 2018/02/19 16:48:24.409 | INFO [SyncWorker<0000275P 1 of 1>] [CatalogVersionSyncMaster] 1. pass, 80472 (+1337) of 103476 items processed (77 %), 19.62 items/sec, 1 (+1, deadlocks:0) items dumped.

4, 2018/02/19 16:51:24.441 | INFO [SyncWorker<0000275P 1 of 1>] [CatalogVersionSyncMaster] 1. pass, 84692 (+1243) of 103476 items processed (81 %), 20.71 items/sec, 1 (+1, deadlocks:0) items dumped. INFO | jvm 1 | main |

5,jvm 1 | main | 2018/02/19 16:52:10.624 | WARN [INT-2,hybris-broadcast,hybrisnode-0] [FD] hybrisnode-0: I was suspected by hybrisnode-1; ignoring the SUSPECT message and sending back a HEARTBEAT_ACK STATUS | wrapper | main | 2018/02/19 16:52:10.624 | Pinging the JVM took 17 seconds to respond.

6[mERROR [Timer-2,hybris-broadcast,hybrisnode-0] [JDBC_PING] JGRP000145: Error updating JDBC_PING table [m com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry XXXXXX-YYYY-ZZZZ-1111-22222a-hybris-broadcast for key 'PRIMARY' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_60] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_60] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_60] at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_60] at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.Util.getInstance(Util.java:360) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2077) ~[mysql-connector-java-5.1.33.jar:5.1.33] at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2062) ~[mysql-connector-java-5.1.33.jar:5.1.33] at org.jgroups.protocols.JDBC_PING.insert(JDBC_PING.java:364) ~[jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.protocols.JDBC_PING.writeToDB(JDBC_PING.java:210) [jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.protocols.JDBC_PING.writeOwnInformation(JDBC_PING.java:193) [jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.protocols.JDBC_PING$InfoWriter.run(JDBC_PING.java:514) [jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:563) [jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:495) [jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:400) [jgroups-3.6.11.Final.jar:3.6.11.Final] at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:260) [jgroups-3.6.11.Final.jar:3.6.11.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]