cancel
Showing results for 
Search instead for 
Did you mean: 

Media and MediaContainer impex performance issue.

Former Member
0 Kudos

Hi,

Importing media is taking too long up to 3 hrs in my case,can any one please suggest the solution for this problem. This is happening for me after migrating from 4.5.12 to 5.5.1.2 . We have around 0.3 million records and imp-ex is processing at 100 items/s at an avg. In old version it processes the same amount of record in 1000 items/s .

We are recently working on migration of hybris 4.5.12 to 5.5.1.2 . We are using same impexes of 4.5.12. But the import time of media data is increased by 10 times. We did not do any change. Sample impex is :

$catalogVersion=catalogVersion(catalog(id[default='']),version[default=''])[unique=true,default=''] $medias=medias(code, $catalogVersion)

"#% impex.test.feeds.context.MCImportContext mcContext = new impex.test.feeds.context.MCImportContext ();"

INSERT_UPDATE MediaContainer;$catalogVersion[pos=300];qualifier[unique=true, pos=301];mediaType[pos=302]; "#%beforeEach: mcContext.prepareMediaContainers(line);" "#% impex.includeExternalData(new FileInputStream(System.getProperty(""HYBRIS_DATA_DIR"") + ""/feed/incoming/IMAGE_CSV.csv""), ""utf-8"", new char[]{','}, 0, -1);" "#%beforeEach:end"

"#% impex.info(""> Time Taken to import Media Containers >""+((System.currentTimeMillis()-startTime)/1000)+"" Sec"");"

This import is very slow with latest hybris version.

$catalogVersion=catalogVersion(catalog(id[default='']),version[default=''])[unique=true,default=''] $medias=medias(code, $catalogVersion)

"#% impex.test.feeds.context.MCImportContext mcContext = new impex.test.feeds.context.MCImportContext ();"

------------------------------------------------------------ CREATING MEDIA

% startTime = System.currentTimeMillis();"

INSERT_UPDATE Media;$catalogVersion[pos=300];code[unique=true, pos=301];mediaFormat(qualifier)[pos=302];url[pos=303];mediaContainer(qualifier, $catalogVersion)[pos=304] "#%beforeEach: mcContext.prepareMedia(line);" "#% impex.includeExternalData(new FileInputStream(System.getProperty(""HYBRIS_DATA_DIR"") + ""/feed/incoming/IMAGE_CSV.csv""), ""utf-8"", new char[]{','}, 0, -1);" "#%beforeEach:end"

"#% impex.info("">> Time Taken to import Media : ""+((System.currentTimeMillis()-startTime)/1000)+"" Sec"");"

Above one also take too long to execute. it is 10 times more than old hybris version.

Please , help us with the possible cause and solution.

Regards, Jeejo

Accepted Solutions (0)

Answers (2)

Answers (2)

andyfletcher
Active Contributor
0 Kudos

Is this possibly related to my question here? https://answers.sap.com/questions/12754721/modification-timestamp-updated-when-no-changes-mad.html

We experienced similar problems with impex now always sending update queries to the db in v5.5, even if there is no change in the data. I suspect you'll find that the extra time is being taken up with db updates that just update the modified time and hjmpts field.

former_member190306
Participant
0 Kudos

not bad.... 100 items/sec even with some beanshell code before each import... not bad 😉

But to come back to your question: I dont know exactly what could be the reason, but there have been some dramatic changes about the general media handling between 4.5.x and 5.5.x E.g., since 4.6 medias itself are not stored within a single directoy but in a btree directory structure. With 5.0 MediaStorageStrategy was invited, etc.

Maybe some of these circumstances are the reasons for a slower performance. Can you do a profiling, where most time is spend?

Former Member
0 Kudos

Thanks , JDBC query is taking very long for new version. earlier query takes 1ms. New one take 150 ms. We are using same version of oracle. Queries generated are exactly same other than PK values .

Former Member
0 Kudos

One more thing, I am not using any specific MediaStorageStrategy as in my case all medias are present in a different location and I am just loading the URL in hybris . Could this be cause of the issue ? Do I need to write any new MediaStorageStrategy for this kind of situation ?

ArthurPerry
Employee
Employee
0 Kudos

Are there any environmental differences between your 4.5.12 instance and the 5.5.1.2 instance? Are appropriate database indexes created on the media tables?

Former Member
0 Kudos

Both are exactly same . Infact I used same schema .

Former Member
0 Kudos

Only JAVA is different as hybris 5.5 use JAVA 8