cancel
Showing results for 
Search instead for 
Did you mean: 

SOLR Indexing running for 2 days

Former Member
0 Kudos

We have SOLR running on the same server as App servers of Hybris. We did data migration for products and started the SOLR indexing. It is running for more than 2 days and not able to complete. the system has 12 GB RAM and has 2 CPU allocated. Let me know if anybody encountered this or optimized SOLR to run faster.

Data volume is not high as the total products is less than 200K.

Former Member
0 Kudos

: What happened next and how did it get resolved. I'm facing the similar issue and your inputs are appreciated. Thanks !

Accepted Solutions (1)

Accepted Solutions (1)

Former Member
0 Kudos

The most likely cause for the indexing to take so long is some custom code that is inefficient during the export... mostly likely some custom value providers. Try profiling the export and also look for long running queries using JDBC logging

Answers (6)

Answers (6)

0 Kudos

Enable JDBC logging and see which query is taking more time. Most probably some of the quries might have bad cache plan causing increased response time

Former Member
0 Kudos

Mostly when this happens it's either too many attributes being indexed, or a lot of products, or more likely a combination of both.

The Number 1 reason for this is indexing a large number of classification attributes as facets.

On a project recently, we had a customer with 1500 attributes. We got it down from 12 hours for a relatively small product set to about 10-15 minutes.

The problem is that a fairly complex FlexibleSearch query with multiple joins is required to get the data from a classification attribute. And if you have a large number of attributes, you are executing it that number of times PER PRODUCT.

The key thing when you have a large number of attributes is to switch from field-level value providers to a model-level value provider.

In v5.6 onwards, there's new functionality that lets you group together field-level value providers so the query happens only once (e.g. for classification attributes) - so moving to 5.6 might help, if it's a viable option.

alimouni
Explorer
0 Kudos

Hi ,

what is this new feature, do you have an example to share with us?

Thanks in advance

0 Kudos

In Hybris 5.3, we've improved SOLR indexing by 4 times by implementing our version of CommerceClassificationPropertyValueProvider getting features from FeatureList and using ClassificationAttributeValueModel instead of JALO.

But we have around 100 facets based on classification attributes.

Former Member
0 Kudos

Brendan Your point is very valid as we trimmed down the value providers by removing unwanted items to be part of the value provider.

Former Member
0 Kudos

SAP recommends stand alone SOLR instance for better performance. We setup a stand alone instance with 32GB RAM.

Former Member
0 Kudos

Hi Ramanathan,

Please try to configured SOLR to commit every let say 5 minutes. Look at the solrconfig.xml. There are several directives related to committing changes but you should not be committing after each record update. Either commit after every 100-200 records or commit every 5 minutes. This is especially important during a reindex of all data.

Solr performance Factor:

A major driving factor for Solr performance is RAM. Solr requires sufficient memory for two separate things: One is the Java heap, the other is "free" memory for the OS disk cache.

It is strongly recommended that Solr runs on a 64-bit Java. A 64-bit Java requires a 64-bit operating system, and a 64-bit operating system requires a 64-bit CPU. There's nothing wrong with 32-bit software or hardware, but a 32-bit Java is limited to a 2GB heap, which can result in artificial limitations that don't exist with a larger heap.