Datahub composition performance

[Former Member]

Hi, we have 2 datahub nodes in our datahub cluster running with IN_MEMORY mode. Publications are ok but there many pending compositions in our pools. Do you suggest to increase composition.page.size and composition.action.size? How will affect other things? Or what can we do to make it better?

[Slava]

It's possible many PENDING compositions are caused by small chunks of data being loaded frequently. Each load schedules a composition that is PENDING while other composition/publication is happening in the pool. So, it's possible changing those configuration properties will not help.

You can check, it's very possible that only one of those PENDING compositions than has data items to process all others finish empty (do not have loaded data or compose 0 items). Or it may not be exactly 0 but distribution of composed items is disproportional: one composition creates many items from several loads, others are either empty or process much fewer items/loads.

If so, this is not really a performance problem.

[Former Member]

i couldnt check the item counts, i count match pending CompositionActions with CanonicalItem itemtypes but here is the picture for performance. it coudnt exhaust pendings and item proceess times are about 20-30 seconds while before it was 0-1 seconds

[Former Member]

also other metrics for GLOBAL pool