XCrawler implementation ...

[thilo_brandt]

Hi together,

is it possible to implement own xcrawlers for 3rd party search engines? If yes, how should I start ?

Regardms

Thilo

[Former Member · Accepted solution]

Hi!

Huuiih, complex question.

You probably would not want to reimplement the crawlers, as you can use them to provide their results to different components than the indexmanagement.

Therefore you must simply implement a crawler result receiver. Then you start the crawlers and tell them to provide their results to your result receiver.

Your result receiver will consist of two parts: the first one is the result receiver factory which the crawler calls to instantiate the actual result receiver - that is the second part.

This sounds more complex than expected, but it is needed for crawlers that survive J2EE downtimes. The crawler memorizes the class name of the result receiver factory and re-instantiates it after the restart. In case you run multiple crawlers you can implement just one factory which provides the different result receivers for the different crawlers.

For the factory you have to implement IXCrawlerResultReceiverFactory.

The result receivers usually implement IXCrawlerPushedDeltaResultReceiver or IXCrawlerPushedResultReceiver depending on if the crawler runs in incremental update mode or not.

You start the crawler via IXCrawlerService.runCrawlerTask().

Kind regards, Dirk

[Former Member]

Hi Thilo,

I also think that developping your own crawlers would be complicated. Perhaps you can use our crawler and try to implement some additional methods in your search service.

When you are able to attach a datasource to your index, the crawlers are started automatically, but you need to implement methods like internalIndex etc.

Regards,

Achim