Grounding AI Models in SAP BTP: Enhancing Accuracy and Context

[thiago]

From automated Grounding to in-database embeddings

This is a summary of the business use case I presented together with my colleague @amagnani37 in this session: Grounding techniques in SAP BTP - From automated grounding to in-database embeddings

Big shotout to colleagues in the Product Management Team for their great support in helping us develop this content: Christoph Morgen, Markus Fath and Mathias Kemeter! Thank you guys!
@ChristophMorgen, @mfath, @mkemeter

Here we land several concepts into something concrete, and for that we have designed a proof of concept that we call Smart Advisory Companion.

This is not a product, it’s just a prototype that we use to illustrate how to implement those services and concepts introduced before and we hope it might not only inspire you but also be of great help as you have access to the working code through the SAP-Samples repository here.

The use case is the typical customer support team, that most of you can relate. We have John the support engineer, and Mary, his manager.

John works taking customer incidents and queries and questions in general, that he answers, or troubleshoots if it’s an issue, or just provide guidance if that’s the case.

Then we have Mary who manages John and his colleagues, the whole team of engineers, to ensure customer satisfaction and optimal support processing.

What they have in common is that both need insights to help on their daily tasks.

John to quickly resolve customer’s queries with high quality standards, and Mary, to identify trends and take better informed decisions for the team.

Let’s talk about John first and see how we can improve his daily tasks.

So, he gets incidents and questions from customers through a CRM running on SAP HANA Cloud. But that’s not enough for him. He struggles a lot when searching the knowledge base for past cases related to the one at hand, and it takes a lot of time to find the right insights.

Let’s see a quick demo on his struggle:

And, back to our problem statement, we want to help John solving it, and the technology introduced earlier today could be useful for that.

So what we did is that we vectorized and stored the embeddings of the knowledge base texts so John can now run meaningful searches instead of just exact matches.

Let’s see how his search experience is now improved.

So, to make this smart search possible, we are using only the recently released Text Embedding Service on SAP HANA Cloud, and nothing else. Meaning the architecture here is very simplified, because it allows us to create the embeddings without calling external services.

To illustrate it better, here you get your unstructured documents with text data, or your text data in a table for example, which is the case here for our support engineer.  Then we use the embedding model to get them embedded into vectors.

The good news is that, as I said, this is now a built-in feature part of SAP HANA Cloud, through the native HANA Text Embedding Service. So, no need to call external services, and it comes with extra benefits that we will see soon, when we discuss the features for the support manager, Mary, such as:

• Reduced Latency: Local computation eliminates the need for network calls, significantly reducing latency.
• Enhanced Security: Data remains within the database, enhancing security by minimizing exposure.
• Cost-Effectiveness: Utilizing local hardware can be more cost-effective than relying on external services, especially for large-scale operations.  

The service makes use of the SAP HANA Natural Language Processing functions which provides not only the vector embeddings but also text analysis functions. Note that in this blog post we cover only the vector embedding and some algorithms that can be leveraged with embeddings such as clustering.

And to do so we need to have both NLP and Script Server activated for our tenant as you can see below.

The model supported here is called SAP_NEB and is based on the MT5 architecture, which is a powerful language model.

It can process up to 256 tokens at a time when using SQL VECTOR_EMBEDDING.

If you use PAL embedding functions, you have more flexibility, with an option to go up to 1024 – but the default is of 256 tokens too, you need to explicitly define more if you need so.

Quick but important remark: make sure you have the user assigned with the "AFL PAL NLP" Execute Role, so you can run these PAL functions without any issue:

Code Deep Dive - Smart Search

Now, how can you implement this smart search using the in-database text embeddings service?

Here is our scenario: John can now run smart searches in the knowledge base to help him finding great insights to help answering customer queries.

Let's imagine this is the table structure where the CRM stores the incidents and queries from customers.

The column TOPIC is the text coming from customers. 

And the column SOLUTION is the text provided by the expert with the answer for the customer’s question, or issue.

To provide that feature to John, we create the columns to convert and store the embeddings from the texts, for both TOPIC and SOLUTION columns.

Please note we are using statement “GENERATED ALWAYS AS” and the function VECTOR_EMBEDDING.

Function VECTOR_EMBEDDING works with these arguments (see full documentation here😞

So, the “GENERATED ALWAYS AS” statement allows us to have the embeddings automatically created as soon as the TOPIC and SOLUTION texts are inserted or updated.

Here you can see the embeddings automatically generated after an insert:

The embeddings CAN NOT be manipulated when using this approach. But why you would, right?

Well, if you have a good reason, and you need to create them manually, another approach that can be used is to just create the columns with type REAL_VECTOR without the automatic embeddings generation:

And in this case you need to convert, insert and update their values manually. Let's see how to do so.

The PAL Embeddings class from the hana_ml TEXT module will help us with that task:

from hana_ml.text.pal_embeddings import PALEmbeddings

We first need to retrieve the records that we want to generate embeddings for, it is, in our case for those with no embeddings.

Then we get the texts we want to generate the embeddings, in this case the columns TOPIC and SOLUTION:

Then we call the method “fit_transform” which is part of the PAL Embeddings class, to convert the input text into embeddings. The input data is a HANA Dataframe with the result of the previous SQL query containing the texts to be converted - the text columns which we indicate as the target in the method call.

To finally insert the results into our newly created columns TOPIC_EMBEDDING and SOLUTION_EMBEDDING.

We retrieve the resulting embeddings by using the prefix VECTOR_COL_ followed by the target column name defined in the fit_transform call, as you can see here (lines 82 and 83 above):

embedding_records[0]["VECTOR_COL_TOPIC"],
embedding_records[0]["VECTOR_COL_SOLUTION"],

Ok, now that we have the embeddings persisted in their respective columns, we want to improve John’s search experience. And we need to compare apples to apples.

So, we need to take John’s question and vectorize it with the same function before comparing to the existing embeddings in the database:

We then convert using the VECTOR_EMBEDDING function and we compare with existing ones using the COSINE_SIMILARITY function.

That’s the same function as before—VECTOR_EMBEDDING. I’m also using the same model.

John is pleased with his enhanced experience, enabling him to deliver higher quality support and expedite assistance for his customers.

Now, let's consider Mary's perspective.

How can we leverage those embeddings so she can identify trends and take better informed decisions?

For her, we implemented few Analytics Features, leveraging the PAL library.

Since the implementation of the Analytics Features for Mary involves more complex capabilities, let's first review the solution architecture of the prototype here.

The app is composed by a Fiori UI5 front-end as you could see in the demos, that relies on Python micro services to consume SAP HANA Cloud services, both for persistency layer and PAL functions. For – what we call “the smart search” - implemented for John, we used the Vector Processing capabilities that you see on the bottom right corner.

But as you can see in the architecture, there’s a lot more that we could do building Embedded AI applications on SAP HANA Cloud that it's worth pausing here a bit and have a look at them.

The out-of-the-box Database AI engine on SAP HANA Cloud gives us access to a range of powerful capabilities, including classification, regression, and time series scenarios. We also get massive data-parallel machine learning, the ability to extract semantic insights from text using native embedding models (which is what we are focusing on in this webinar today), and machine learning on vector data for tasks like clustering, classification, and regression - and that’s just the beginning.

For each scenario listed in the Database AI Engine, we can leverage a number of features and different algorithms.

We, of course don't intend to explain each feature in detail here, but I highly recommend you to go and check them out to explore what you can do without the need of consuming external services, at in-memory processing time speed.

In our case here, for our prototype code and demo purposes, Mary will greatly benefit from a few key features you see here that we're leveraging in our prototype, and we will discuss now.

So let’s focus on those we applied to our use case: we have already seen the text embedding processing for John, now we will see how to help Mary with Cluster Analysis (with K-Means, TSNE and Categorical PCA).

Let’s get started with the Clustering Analysis.

Clustering Analysis

Mary got two main needs: identifying trends and making better-informed decisions

We want to give her a Cluster Analysis view of incidents and queries so she can spot trends — like emerging topics and technologies that might not have enough coverage within the team.

Let’s see what we can leverage with SAP HANA Cloud for our manager Mary.

With this capability and a proactive assessment, Mary can now train her team in advance, preventing a flood of incoming requests on totally new topics. We could even consider integrating this with SuccessFactors Learning.

Let's now jump into the code and see how the Clustering Analysis implementation we've just demonstrated looks like.

We first need to have at least two tables here, one to store the clustering results by incident and another for the cluster master data.

This one will persist incidents sorted by clusters that can be represented in a cartesian graph:

 And this one will store the master data for each resulting cluster, like its ID and description:

Ok, tables created. Let’s now see what classes we are going to use to implement the clustering view:

You can see we are importing three classes here: the Categorical Principal Component Analysis, or CAT PCA. This is a statistical technique used to reduce the dimensionality of data while preserving as much variability as possible. This is useful for simplifying complex datasets and making them easier to analyze.

The second is the TSNE which stands for t-Distributed Stochastic Neighbor Embedding. It is a machine learning algorithm used for dimensionality reduction, in our case well-suited for visualizing high-dimensional data in a two or three-dimensional space. It helps in understanding the structure of the data by clustering similar data points together.

And finally the third and most popular: Kmeans – which is a popular clustering algorithm used to partition a dataset into a specified number of clusters. Each data point is assigned to the cluster with the nearest mean.

Business-Driven Categorization

Next we have the Business-Driven Categorization, which is another great feature we unlocked with the use of embeddings.

The K-means cluster analysis we just saw gives Mary great insights focusing on technical matters, such as AI, Database, Integration, and so on so forth.

However, she also needs a Business-Driven categorization. With the Embeddings, we can offer her a personalized custom categorization that will allow her to match company priorities and goals. Let’s see it in this video:

Let’s see how we developed this Custom On-Demand Categorization functionality.

So here’s the business flow, Mary wants to have a custom categorization so she not only report it to her managers but also understands her resources workload as per company priorities and then, for example, reroute or reassign resources by priorities.

First, we need create two tables that will hold the categories and the results for the categorization.

Here’s one to store the Categories defined by the manager, with the automatic embeddings’ generation in place for the category description, as I mentioned during the demo above.

This is the Projects by Category table to store the final categorization results.

Ok, so we first insert the manager’s provided categories into the table, which automatically generates the embeddings.

Using COSINE_SIMILARITY we compare each incident record in the incidents table with the manager’s categories and store the results in a HANA data frame.

To finally save the most similar into the Projects by Category table.

Pretty straight forward.

WRAP-UP

We’ve seen a comparison between several methods for Grounding and an overview of the SAP HANA Cloud Text Embedding Service, which uses the SAP_NEB model and can be consumed either though SQL or PAL function.

With that we got not only the benefit of running vectorization without the need to rely on external services but also we could see how to take advantage of those embeddings for ML processing using few examples.

Those are only basic examples so you can get inspired and go through that full list of features I've shown earlier.

We will integrate these features, that we develop for John and Mary, into Joule and show you how to do so, so you can get inspired and integrate your solutions features there, into Joule, too.

So, you know, don’t miss out and register now to the upcoming webinars:

Explicit knowledge representation and reasoning with Knowledge Graphs

Session 01 : 06th May 2025, 09:00 AM CET - 11:00 AM CET
Session 02 : 07th May 2025, 10:00 AM EST - 12:00 PM EST

Amplify Joule's power for your enterprise needs

Session 1: 17th June 2025, 09:00 AM CET - 11:00 AM CET
Session 2: 18th June 2025, 10:00 AM EST - 12:00 PM EST

Stay tuned!