data hub cpu to high

[former_member1208351]

While trying to export data (user, products or orders), the data hub processs takes more than 90% cpu on source database causing a bad performances and even crashes. I cannot use the data hub to migrate my data in those conditions if the database cpu is too high. What can be the solution? splitting the jobs and run them manually would be painful . Is there a clean solution for the data hub to run by steps or to make it less greedy?

[Marko_salonen]

And for best performance, data hubs database should be in its own node and not share with any other instance

[Former Member]

I think you need architectural changes in the landscape, Datahub should be on its own host, secondly Datahub shouldn't be talking to Nodes where store front is running.

Thanks Sagar

[former_member333910]

Olivier, a few comments:

• Data Hub distributes processing across the available CPUs. This is by design and a performance optimization

• You mention "data hub uses more than 90% cpu causing a bad performance for customers while browsing the website". Do you have Data Hub and Hybris Commerce running on the same machine? If so, this is not a recommended deployment scenario. Otherwise, I don't see how Data Hub CPU usage would affect Hybris Commerce.

• Are you following the hardware guidelines for your Data Hub installation? Particularly with regarding to memory, CPU, and HEAP. https://help.hybris.com/6.7.0/hcd/2e17d39b612840648c76690dd2696645.html

[VinayKumarS]

Can you check your database size. Becuase if your database clean up is having some issue then huge amount of data will be residing which leads to an issue in the performance.

Also check if you have added any business logic on conanical items. that also may cause the performance issue on the datahub.

[former_member1208351]

Thanks for the answer, do you know which tables are involved for the cleanup? I mean which tables are cleaned, will be easy to fin which one is not properly cleaned.

[VinayKumarS]

check the conanical and target tables.

[former_member1208351]

As I said data hub database is out of scope for this issue, event with an empty db the process to generate the impex are quite expensive

[VinayKumarS]

Hi Olivier, As per my experience datahub cpu is high when you have more complex logic with database entries. Thats why shwetha and myself suspecting its a database issue. You are pretty much sure its not a database size issue. But can you double check the database table sizes. When you do the clean up job cleanup. Database size should not be high. or may be if you have written custom logic which might be going multiple loops. Any of this may cause higher usage of CPU. Also when you create a discounts that time these discounts also update in the Datahub. Which is also might cause the issue.

[Former Member]

Do you have datahub cleanup properties set in your datahub local.properties? Clean up properties delete the processed raw , canonical and target items. So that the database will be lighter.

[former_member1208351]

yes I do. I don t think that the data hub database is the cause, it seems more that the source database is full busy while creating the impexes for the export. It occurs before sending the request to data hub.

[Marko_salonen]

I would recommend checking if there are some database indexes that could be added. This is usually the case with data hub.

[former_member1208351]

Hi thank you for your answer, do you know where to put them? which tables ? and columns? the data hub tables are already indexed and the tables to export too...

[Marko_salonen]

Hi, i can not really say, depends on your data. We have been adding in both CanonicalItem and RawItem. A DBA would possibly help with this