Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
Showing results for 
Search instead for 
Did you mean: 
Product and Topic Expert
Product and Topic Expert
This blog post is part of a series of technical enablement sessions on SAP BTP for Industries. Check the full calendar here to watch the recordings of past sessions and register for the upcoming ones! The replay of this session available here.

Authors:, amagnani, edward.neveux, jacobtan 

Disclaimer: SAP notes that posts about potential uses of generative AI and large language models are merely the individual poster's ideas and opinions, and do not represent SAP's official position or future development roadmap. SAP has no legal obligation or other commitment to pursue any course of business, or develop or release any functionality, mentioned in any post or related content on this website.


Large Language Models (LLMs) like ChatGPT and GPT-4 have gained significant popularity with emergent capabilities. In this fourth session of the series, we’ll explore the potentials of GPT in SAP ecosystem, and discuss how SAP partners can leverage this powerful AI technology in combination with SAP technologies to accelerate their solution development processes, and create more intelligent solutions on SAP Business Technology Platform (SAP BTP).

Here list the full blog post series of Exploring the potential of GPT in SAP ecosystem:

Exploring the potential of GPT in SAP ecosystem

Part 1 – GPT in SAP ecosystem

Part 2 – GPT as code assistance for SAP CAP and RAP Application development

Part 3 – Prompt Engineering for Advanced Text Processing on Customer Messages

Part 4 – Integrate with GPT Chat API for Advanced Text Processing (this blog post)

Part 5 – Integrate with GPT Embedding API for Customer Message Classification

In our previous blog about we have seen GPT’s advanced text processing capability on customer messages through prompt engineering, such as text summarization, sentiment analysis, entities extraction etc.

In this blog post, we will proceed with integrating the prompts into the Intelligent Ticketing Solution for processing customer messages with GPT through its Chat API. Additionally, implementing the LLM Proxy Service in the architecture diagram.

A glimpse into the Chat API of GPT

Let us start with a look into GPT’s Chat API.

In the playground, we can examine the underlying API request to Azure’s OpenAI Service when submitting a prompt by clicking the "View Code" on the Chat session. Let us consider the example of analyzing the sentiment of input text.

In the following diagram, you can find the API endpoint, and API key to access GPT Chat API. The request body of Chat API consists of a system-role message for the instruction and a user-role message for the user input message, along with other parameters such as temperature, max_tokens, etc.

For more detail about Chat API, please refer to Azure OpenAI Service API Reference.

Using the prompts through GPT’s Chat API in NodeJS with SAP CAP

Next, let us look at some sample code on how to perform the API call using the same sentiment analysis prompt in SAP CAP Node.js, which we have seen previously.

Please refer to gpt-api-samples for the NodeJS samples of using Chat API of GPT-3.5 onward models and Embedding API of text-embedding-ada-002 (Version 2) models in Azure OpenAI Service through four options.

  • langchain: A popular open-source framework for developing applications powered by large language models. With LangChain, you can leverage multiple Large Language Models such as OpenAI GPT models or Meta LLaMa models etc.

  • @azure/openai module:  The Azure OpenAI client library for JavaScript is an adaptation of OpenAI's REST APIs that provides an idiomatic interface and rich integration with the rest of the Azure SDK ecosystem

  • azure-openai module: A fork of the official OpenAI Node.js library that has been adapted to support the Azure OpenAI API.

  • Direct http call with axios module: Making a direct HTTP call to the RESTful API of GPT using any HTTP client in Node.js, such as axios, request etc.

Implementing a reusable, LLM-agnostic and configurable LLM Proxy Service

Integrating a single prompt into an SAP CAP application using GPT’s Chat API is quite straightforward.

However, what if you have multiple prompts that need to be integrated?

What if the prompts have dynamic input parameters?

What if you need flexibility, instead of being locked-into a specific Large Language Model?

Therefore, it is a best practice to implement a Large Language Model Proxy Service in your SAP CAP solution with the design principles as follows:

  • It makes sense to have a reusable service that handles all the integrations with Large Language Model APIs, instead of scattering the integration glue code throughout the application.

  • Being Large-Language-Model-agnostic will give you the flexibility to choose a Large Language Model based on your specific requirements.

  • Making prompts configurable through a configuration mechanism, rather than hard-coding them as static and fixed prompts in the code, will provide greater flexibility and ease of maintenance.

In our sample intelligent ticketing solution, we have implemented such a Large Language Model Proxy Service based on these principles. Next, we will explore into its design and implementation details.

Turning the Prompts into a custom generic RESTful API in SAP CAP

In previous blog post about Prompt Engineering for Advance Text Processing on Customer Messages, we have seen all the prompts share the same structure consisting of Instruction, User Input (Input Indicator, Text Message and Output Indicator) and Response in JSON etc.

Take text summarization and sentiment analysis for example, the only differences are instruction and response.

Therefore, it is possible to abstract it as a custom generic RESTful API in SAP CAP illustrated in the diagram below, in which the details of the prompts and LLM API access can be hidden and configured into a JSON configuration file as config.json, accessible through the concept of use_case.

The API only requires two simple input parameters:

  • use_case: text-summarization, sentiment-analysis etc, which can be used to identify its associated prompt and api access info from configuration.

  • text: The text message of inbound customer message to be responded to.

Configuration of LlmProxyService

The diagram below illustrates how use-case(sentiment-analysis) can be used to identify its associated prompt detail and API access information in config.json. In this way, we can make the LLMProxyService configurable.

Service Definition of LlmProxyService

In llm-proxy-service.cds,  a generic action named invokeLLM() is defined, along with its output schema as CustomerMsgResultType, which could be accessed through a generic RESTful API or a direct action call of LlmProxyService outside or within SAP CAP for all the integration with LLM APIs.

Authentication to LLM API

In our sample of LlmProxyService implementation, we have chosen direct http call via axios to LLM API, which makes it LLM-agnostic possible with configuration about LLM API access, such as “api_base_url” and “api_auth”. Certainly, it could be also achieved with LangChain etc.

Let’s have a look at the authentication to LLM API. The implementation details can be found here, which can support both OpenAI and Azure OpenAI Service through configuration api_auth: “bearer-api-key” and api_auth: “api-key”.

Implementation of action invokeLLM() in LlmProxyService

Let’s have a look at the implementation details of action invokeLLM() in LlmProxyService.

Test on Final REST API of invokeLLM

Finally, let us run some tests on the REST API of invokeLLM in LlmProxyService with

use_case - sentiment-analysis

use_case - customer-message-process

Further resources

Wrap up

We have seen that how to integrate the prompts into an SAP CAP NodeJS Application with GPT’s Chat API, even implementing a generic, reusable, and configurable LLM Proxy Service to handle all the integration of LLM APIs and turning the prompts into a custom RESTful API in SAP CAP, leading to the flexibility of selecting LLMs, and ease of maintenance.