In HANA SDS , Kafka Input Adapter Fetching is very slow

[bintu__]

Hi lin.murong

I am using KAFKA json Input Adapter to fetch messages from Kafka Topic. Messages getting fetched successfully.

But I could see messages getting fetched very slowly into HANA SDS Projects due to which lag is increasing at the source system.

I am using below mentioned parameters in my kafka Input adapter.

ATTACH INPUT ADAPTER i_kafka_1 TYPE toolkit_kafka_json_input

TO i_biz_stream

PROPERTIES

kafkaTopicPartition = 'EVENT1' ,

kafkaOffset = 0 ,

kafkaFromBeginning = FALSE ,

kafkaBootstrapServers = 'bk.0001.111' ,

kafkaGroupId = 'hanasds_consumer' ,

kafkaConsumerProperties = 'consum.properties' ,

kafkaPollInterval = 1 ,

jsonColumnMappingFile = '' ,

jsonColumnMappingXml = '' ,

jsonColsMappingList = 'topic,offset,value.content.eventId',

jsonRootpath = '' ,

additionalStreams = '' ,

gdMaxRetryTimes = 1 ,

jsonSecondDateFormat = '' ,

jsonMsDateFormat = '' ,

jsonTimeFormat = '' ,

jsonBigdatetimeFormat = '' ,

jsonCharsetName = '' ,

enableGdMode = TRUE ,

maxPubPoolSize = 100000 ,

useTransactions = FALSE ;

Could you please suggest any other way to increase performance of Input adapter. ?

[RobertWaywell]

How slow is slow in this case? Are you talking 1 event per minute or 10,000 events per second or ??

At a first look the most obvious thing to try would be to use multiple partitions in the Kafka topic and then have the input adapter read from the multiple partitions in parallel.