Understanding Joule - (Step #6) Document Grounding in Joule

[Raja_Gupta1]

This blog is part of the series Understanding Joule in 8 Baby Steps Please check the previous blogs, before proceeding.

 #1 – Introduction to Joule

#2 How Joule Works

#3 Joule Architecture

#4 Joule Interaction Patterns

#5 Joule in SAP SuccessFactors

In this blog, we will dive deep into Document Grounding in Joule.

Let's First Understand What Document Grounding is

Let’s understand this with an example.

If you ask Joule a question which relies on a company's internal information, for example:

“Can I use my private car for my business trip?”

You probably know the answer is in a policy documents, but don’t want to go through documents but just get answer using Joule.

This is exactly what Joule’s document grounding capability can do.

If Joule response this in usual way, by leveraging a LLM, then the response will be a generic one based on general SAP knowledge.

However....

If we feed Joule with company’s specific documents (like internal policies, reports and documents), then Joule can use “document grounding” and give more accurate and relevant answers.

So, what exactly is document grounding?

Document grounding is the process of making a LLM refer to specific documents to generate more accurate and relevant responses.

Instead of relying only on its pre-trained knowledge, the LLM pulls information from external sources to answer questions with real, up-to-date data.

Document Grounding in Joule

The document grounding capability of Joule helps us:

• Upload documents in SAP and third-party repositories
• And help Joule provide more comprehensive responses using information gathered from those documents

How does Document Grounding in Joule work?

Below image shows how document grounding works in Joule.

First customers need to setup the document grounding in Joule:

1. Customers connect their document repositories via their own Microsoft SharePoint.
2. Document Grounding AI Service is setup in SAP BTP.
3. The Document Grounding AI Service takes the relevant customer-owned documents via Microsoft SharePoint and creates so called, vector embeddings.
4. The vector embeddings are stored in SAP HANA Cloud Vector Engine.

Once set up, this process runs once per day to ensure the latest document contents are always available to Joule.

When users can ask a query to Joule and flows work as below:

1. User enter prompt in Joule UI and its send to Joule service on SAP BTP.
2. Joule’s dialog manager sends the user question/prompt to its internal RAG service.
3. The RAG service creates a vector “embedding” of the prompt. The vector embedding in simple words is - the vector of numbers that represents the question. The vector embedding is further used to perform a “vector query” against SAP HANA Vector Engine. The vector query is a kind of mathematical similarity search that returns sections of documents that are relevant to the users question. In this example, it might be sections from an employee manual detailing allowable trip expenses and reimbursement rates.
4. SAP HANA Vector Engine returns the relevant document extract.
5. The RAG service passes the relevant documents extracts back to the Joule Dialog Manager along with information about the source documents.
6. The Dialog Manager appends the user question with the relevant extracts and sends the “enriched” prompt to the LLM.
7. The LLM provide the response, “grounded” with the company-specific data and also appends links to the sources of the enrichment data.
8. The Dialog Manager further sends the response along with links to sources, back to the Joule UI. The user gets a concise, reliable, natural language answer. They can optionally explore the links for quick verification and more details.

Current Scope of the document grounding capability in Joule

• Data repository: Microsoft SharePoint
• File format: PDF, Microsoft Word, HTML (Static),  JPEG,  JPG,  PNG,  TIFF,  TXT
• Document content: Plain text (Tables and images are not supported)
• Language: English
• Content refresh: SharePoint content updated once every day

This is in nutshell - Document Grounding in Joule!

What's Next?

Blog #7 - Joule for Developers

Meanwhile, you may read the blog Joule, SAP’s Generative AI Copilot