SAP Customer Data Cloud Dataflow Custom Script

[S_Sarkar]

While working with SAP Customer Data Cloud Dataflow Custom Script. What is way to move to the next record inside record.evaluate component. Let me share my expectation,

If record.profile.firstName != "Jack", Then the current record value will be assigned to the next record value. Suppose there are three records: record1, record2, record3. If the record1 does not have Jack in first name, then I want to move to record2 (i.e. record = record2).

[samuelyang · Accepted solution]

Hi @S_Sarkar Custom data fields, if not encrypted, are also indexed and searchable in CDC, you may try it out in your query and check if the query performance is acceptable? This way you can skip those records in your dataflow. 

[S_Sarkar · Accepted solution]

Thanks a lot samuelyang, this information is helpful indeed. Seems like there is no other way option to get rid of != in the search query. Rather the imported data will be taken in smaller chunks to avoid perfomance issue..

[KunalBansal]

Hello @S_Sarkar 
AFAIK, in CDC dataflow,  it's not possible to SKIP a record using the record.evaluate function and move to the next one! What's your exact use-case, what type of Business Usecase you're trying to achieve?
Feel free to reply for further queries.

Thanks & Regards,
Kunal Bansal, SAP CX SME
https://www.linkedin.com/in/thekunalbansal

[S_Sarkar]

Thanks for your reply, I want to process only those records which don't have certain value in the custom field. Further, I can ignore those values by using != in the accounts.search API query. But I am afraid != for a custom field may make the search slow. So I want to pull records and further filter it in record.evaluate with an if condition. Could you please suggest any option? or is it okay to have != in the search query for a custom field. I assume custom fields are not indexed in CDC internally.

[KunalBansal]

Hi @S_Sarkar, Indeed performance will be impacted when using != search query. Is this a one time dataflow or you want to schedule it on a freuency? What's you current data volume in millions?