cancel
Showing results for 
Search instead for 
Did you mean: 

Correct way to provision the destination for calling LLMs from SAP CAP applications GenAI Hub?

abhshia
Explorer
0 Kudos
376

What is the correct way to provision a BTP destination for calling LLMs from SAP CAP applications on BTP to GenAI Hub viaSAP AI Core?

I'm using SAP CAP plugin

 

 

 

npm install cap-llm-plugin

 

 

 

Updating package.json in my CAP project with following:

 

 

 

"cds": {
    "requires": {
      "destinations": true,
      "cap-llm-plugin": true,
      "gen-ai-hub": {
        "anthropic--claude-3-sonnet": {
          "destinationName": "GenAIHubDestination",
          "deploymentUrl": "/v2/inference/deployments/d3b...6575",
          "resourceGroup": "default",
          "anthropic_version": "bedrock-2023-05-31",
          "max_tokens": 4096,
          "modelName": "anthropic--claude-3-sonnet"
        }
      },
      "GenAIHubDestination": {
        "kind": "rest",
        "credentials": {
          "destination": "ai_core_dest_4hybridmode",
          "requestTimeout": "300000"
        }
      }
    }
  }

 

 

 

Next, I configured a Lite destination instance and  configured a new destination titled `ai_core_dest_4hybridmode` . Following tutorial here and used credentials from my AI core instance.

https://github.com/SAP-samples/codejam-cap-llm/tree/main/exercises/05-explore-destination-service#un...

Since I am developing locally in Hybrid mode, I am binding the local CAP deployment to my destination in the cloud using which creates an entry in `.cdsrc-private.json` in the root folder of my project.

 

 

 

cf create-service destination lite cpapp-destination
cf create-service-key cpapp-destination cpapp-destination-key

 

 

 

 

 

 

 

cds bind -2 cpapp-destination:cpapp-destination-key

 

 

 

Next, based on Akash's input - used the following data for BTP destination:

Name: ai_core_dest_4hybridmode
Description: SAP AI Core deployed service (generative AI hub)
URL: <AI-API-INSIDE-AI-CORE-SERVICE-KEY>
Type: HTTP
ProxyType: Internet
Authentication: OAuth2ClientCredentials
tokenServiceURL: <TOKEN-SERVICE-URL-OF-AI-CORE-SERVICE-KEY>/oauth/token
clientId: <YOUR-CLIENT-ID-OF-AI-CORE-SERVICE-KEY>
clientSecret: <YOUR-CLIENT-SECRET-OF-AI-CORE-SERVICE-KEY>
# Additional Properties:
URL.headers.AI-Resource-Group: default # adjust if necessary
URL.headers.Content-Type: application/json
HTML5.DynamicDestination: true

Note the trailing `/oauth/token` at the end of the tokenServiceURL.

Use 

 

 

 

cds watch --profile hybrid

 

 

 

 

 

For reference my service.js is:

 

const cds = require('@sap/cds')
const logger = cds.log('llm-invoke')
module.exports = cds.service.impl(
    function () {
        logger.log('-------------')
        logger.log('-------this is an anonymous function invoking llm via GenAI hub------')
        logger.log('-------------')


        this.on('invoke', async (req) => {

            let messages = [{
                "role": "user",
                "content": "Hello, Claude"
            }]

            // Reference: https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/consume-generative-ai-models-using-sap-ai-core#aws-bedrock
            // Reference: https://github.com/ashivadi/btp-gen-ai-hub-sdk-samples/blob/main/docs/notebook-samples/5-invoke-bedrock-models-via-genaihub.ipynb
            const chatCompletionReqPayload = {
                "anthropic_version": "bedrock-2023-05-31",
                "max_tokens": 512,
                "messages": messages,
            };

            const chatModelConfig = cds.env.requires["gen-ai-hub"]["anthropic--claude-3.5-sonnet"];
            /* just for reference, chatModelConfig from package.json is: 
            {
                "destinationName": "GenAIHubDestination", 
                "deploymentUrl": "/v2/inference/deployments/d3bx.....58475",
                "resourceGroup": "default",
                "anthropic_version": "bedrock-2023-05-31",
                "max_tokens": 4096,
                "modelName": "anthropic--claude-3-sonnet"
            }
            */
            const llmBedrockPluginViaCAPLLM = await cds.connect.to("cap-llm-plugin");
            let result = await llmBedrockPluginViaCAPLLM.getChatCompletionWithConfig(chatModelConfig, chatCompletionReqPayload);

            logger.log('-------Result------')
            logger.log(result)
            logger.log('-------------')

            return result;

        });


    }
)

 

 

Note, 

It is expected that you get a 404:Not Found at the check Connection in my destination? My configuration works even with the check saying 404.

abhshia_0-1732060248826.png

 

 

 

View Entire Topic
AkashAmarendra
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hello,

The error looks like the CAP application fails to connect to the destination. Can we check if the destination service is bound to the CAP application. Here is a sample application with steps to replicate the usecase which includes steps to configure the destination and bind it to the CAP application : https://github.com/SAP-samples/cap-llm-plugin-samples/tree/main/samples/rag-quickstart-app

Happy to assist further if the issue persists.

Thanks

abhshia
Explorer
0 Kudos

Thanks Akash, thank you for the repo - that was exactly the repo I was using to construct my `package.json` file. I updated the original post with the steps to bind my CAP project in hybrid mode. Somehow, there is some type of misconfig. Have you seen this error before?

 

 

statusCode: 502,

reason: {

message: 'Error during request to remote service: Failed to build headers. Caused by: The destination tried to provide authorization tokens but failed in all cases. This is most likely due to misconfiguration.\n' +

'Original error messages:\n' +

'Could not verify the provided CSRF token because no token was found to compare.',

 

AkashAmarendra
Product and Topic Expert
Product and Topic Expert

Hello,

The new issue seems to an issue with authorization. Can you please check the destination configured 

Name: aicore-destination
Description: SAP AI Core deployed service (generative AI hub)
URL: <AI-API-INSIDE-AI-CORE-SERVICE-KEY>
Type: HTTP
ProxyType: Internet
Authentication: OAuth2ClientCredentials
tokenServiceURL: <TOKEN-SERVICE-URL-OF-AI-CORE-SERVICE-KEY>/oauth/token
clientId: <YOUR-CLIENT-ID-OF-AI-CORE-SERVICE-KEY>
clientSecret: <YOUR-CLIENT-SECRET-OF-AI-CORE-SERVICE-KEY>
# Additional Properties:
URL.headers.AI-Resource-Group: default # adjust if necessary
URL.headers.Content-Type: application/json
HTML5.DynamicDestination: true

Please replace the above parameters using the ai core service key. Also please make sure that the appropriate roles are assigned on BTP (ai core specific roles).

Thanks!