cancel
Showing results for 
Search instead for 
Did you mean: 

cluster threads using 50% of CPU

Former Member
0 Kudos

Hello,

Our environment consist of a loadbalancer and two Hybris servers (App1 and App2) configured in clustered mode connecting to common database server. At any given time java process is utilizing 50% of CPU load irrespective of any external load.

When we run top command in thread mode(SHIFT+H) there are 2 threads utilizing 25% of CPU load each. This threads are of cluster property.

There is no issue in cluster configuration. When cluster mode is set to FALSE the CPU utilization drops to 1%. Is this behavior normal or is there any issue in cluster environment??

former_member537989
Contributor
0 Kudos
  • What is your OS?

  • "At any given time" - so you were monitoring stack traces long enough? Are these threads always in "Running" state?

  • Are you sure you've identified the CPU consuming threads correctly?

Former Member
0 Kudos

Hi ,

Please find OS details below,

[ihpadm@SAPHYBRISAPP1 ~]$ lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 6.6 (Santiago) Release: 6.6 Codename: Santiago

Yes. When application is running in cluster mode mentioned threads are always in RUNNABLE state.

Also when we disabled cluster mode CPU utilization drops to 1% and these threads are not running.

Accepted Solutions (0)

Answers (3)

Answers (3)

Former Member
0 Kudos

Hi All,

Thanks for your responses.

I have found the root cause of the this issue. If you have configured hybris server in cluster mode and you are facing the same issue of high CPU utilization than make the changes in following property.

local.properties => under cluster configuration,

cluster.broadcast.method.unicast.sync.nodes.interval=

If the value of above property is 0 then set it to 1. It will solve the high utilization problem.

I think when this property is set to 0 both the cluster & cache threads mentioned earlier are always in running state reserving two cpu cores receptively which was causing 50% utilization.

Former Member
0 Kudos

Hello Mandar,

I have faced same problem, Can you please share your cluster whole configuration set in local.property?. It will help me :)

Former Member
0 Kudos

Hybris version 5.2. Clustering method is jgroups. No network latency as such. No cluster warning in cluster logs.

CPU configuration is 4 cores Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz.

Kindly refer following top command output in thread mode.

top - 11:59:50 up 62 days, 18:27, 1 user, load average: 1.29, 1.31, 1.27 Tasks: 510 total, 3 running, 507 sleeping, 0 stopped, 0 zombie Cpu(s): 27.6%us, 14.3%sy, 0.0%ni, 50.0%id, 0.0%wa, 0.4%hi, 7.6%si, 0.0%st Mem: 32879416k total, 28008416k used, 4871000k free, 1742212k buffers Swap: 69206008k total, 109188k used, 69096820k free, 11398012k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21097 ihpadm 20 0 28.3g 12g 30m R 25.0 39.6 929:03.21 java 21098 ihpadm 20 0 28.3g 12g 30m R 24.7 39.6 937:46.80 java

Following are the two threads consuming 25% of CPU each.

"UnicastSyncNodesThread" daemon prio=10 tid=0x0000000001285800 nid=0x526a runnable [0x00007fc5f658e000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.send(Native Method) at java.net.DatagramSocket.send(DatagramSocket.java:676) - locked (a java.net.DatagramPacket) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod.sendPacket(UnicastBroadcastMethod.java:538) - locked (a de.hybris.platform.cluster.udp.UnicastBroadcastMethod) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod.send(UnicastBroadcastMethod.java:349) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod.send(UnicastBroadcastMethod.java:288) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod.sendMyNodesSyncMessagesIfEnabled(UnicastBroadcastMethod.java:320) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod.access$1(UnicastBroadcastMethod.java:303) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod$SyncNodesThread.run(UnicastBroadcastMethod.java:497)

"Unicast Distributed Cache Server" daemon prio=10 tid=0x0000000001282800 nid=0x5269 runnable [0x00007fc5f668f000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked (a java.net.PlainDatagramSocketImpl) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145) - locked (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:786) - locked (a java.net.DatagramPacket) - locked (a java.net.DatagramSocket) at de.hybris.platform.cluster.udp.UnicastBroadcastMethod$UnicastServerThread.run(UnicastBroadcastMethod.java:693)

former_member602476
Active Participant
0 Kudos

What is your hybris version, what is the clustering method, what is the network latency between the 2 nodes, any cluster warnings in the logs? Can you reproduce with standard accelerator?