How Metadata Enrichment can enhance SAP HANA Cloud...

DKK · ‎2024 Apr 29

Introduction

In the ever-evolving landscape of data-driven businesses, the need for intelligent data applications that can process and analyse vast amounts of data accurately and rapidly has never been more critical. There are already several interesting SAP Blogs regarding the SAP HANA Cloud Vector Engine and Langchain on our community page, such as HANA Vector Engine and LangChain , SAP HANA Cloud's Vector Engine Announcement , Vectorize your Data : SAP HANA Cloud's Vector Engine for Unified Data Excellence , First steps using the Hana Vector Engine with SAP GEN AI and more. On this blog we will focus on how Metadata Enrichment and the SAP HANA Vector Engine, can improve the performance of Retrieval-Augmented Generation (RAG for short) scenarios.There are multiple ways by which we can leverage metadata enrichment into RAG scenarios, but for the purposes of the present case we will employ the OpenAi Metadata Tagger.

Why do I need Metadata Enrichment?

Metadata enrichment can significantly enhance Retriever-Augmented Generation (RAG) scenarios by providing additional, contextual information that helps in more accurate and relevant document retrieval. This information can include basic information such as the document title and page from which the information was retrieved but can also be extended to host even additional information, such as sentiment of the text chunk, topics discussed, summaries etc. By enhancing the documents with additional metadata the retriever's performance can be improved as these additional information can provide search handles and contextual cues that aid the retriever in finding more accurate matches from the corpus.

However, in real-life applications, several documents may need to be processed and undergoing the metadata enrichment task manually can be tedious and time consuming task. Nevertheless, by means of GenAI-based document transformers, we can streamline the metadata extraction process.

The benefits of Metadata Enrichments via a Business Example

In today's digital era, online platforms have become a prominent marketplace for a plethora of products, attracting millions of consumers worldwide. Consequently, these platforms are overwhelmed with customer reviews, which are pivotal for businesses to assess customer satisfaction and product performance.
However, manually analysing these reviews can be work-intensive and time-consuming. This is where sentiment analysis, comes into play.

Sentiment analysis, also known as opinion mining, refers to the use of text analysis, computational linguistics, and machine learning to identify and extract subjective information from source materials. In the context of online marketplaces, this means gauging customer sentiment towards products or services based on their reviews.

Business Cases for Sentiment Analysis of Customer Reviews

Enhancing Customer Experience: The primary objective of any business is to meet and exceed customer expectations. By analysing customer sentiment, companies can gain insights into what customers like or dislike about their products or services. This knowledge can be leveraged to improve product offerings and enhance customer experience, leading to higher customer retention and increased sales.
Competitive Advantage: By keeping an eye on the reviews of competitor products, businesses can identify their strengths and weaknesses. Sentiment analysis can provide a competitive advantage by helping firms understand areas where they can differentiate themselves.
Reputation Management: Negative reviews can tarnish a company's reputation and affect its bottom line. Sentiment analysis allows for real-time monitoring of reviews, enabling companies to promptly respond to negative feedback and manage their online reputation effectively.
Forecasting and Strategic Planning: Sentiment analysis can be used to predict future sales trends based on customer sentiment. This information can be invaluable for strategic planning, inventory management, and decision-making.

As already highlighted, sentiment analysis can bring significant value to modern enterprises, however to render those insights meaningful and actionable from a business perspective, more services (such as systems integration, automation or data and analytics) may often be required, For this reason, on this blog we will also discuss how and which services from the SAP Business Technology Platform (BTP) can help us bridge this gap.

Demo

On our scenario, we decided to analyse the category "All Beauty" from the Amazon Reviews 2023 . We merged the two available DataSets , Reviews and Product Description, in one. The initial Datasets were few hundreds megabytes in size and provided with Json format. Using the Pandas library and SAP HANA ML API we can effectively process these data and load them in HANA through and easy to use pythonic interface.

##Connect to SAP HANA
##Read JSON and 
##Load data to HANA in Batches
import pandas as pd    
file_path=r'./All_Beauty.jsonl'
jsonObj = pd.read_json(path_or_buf=file_path, lines=True, chunksize=1000)
 with pd.read_json(path_or_buf=file_path, lines=True, chunksize=10) as reader:
     for i, chunk in enumerate(reader):
         print(chunk)
         print(type(chunk))
         break
        
         chunk.columns = map(str.upper, chunk.columns)
         display(chunk)
         chunk["TITLE_WITH_TEXT"] = chunk["TITLE"] + chunk["TEXT"]
         chunk['SMALL_IMAGES']=chunk['IMAGES'].apply(lambda x: ', '.join([d['small_image_url'] for d in x]))
         chunk['LARGE_IMAGES']=chunk['IMAGES'].apply(lambda x: ', '.join([d['large_image_url'] for d in x]))
         chunk = chunk.drop(['IMAGES'], axis=1)
         display(chunk)
         print(chunk.dtypes)
         if i==0:
             dataframe.create_dataframe_from_pandas(
                 cc, chunk, chunk.iloc[:, [0,1, 2, 3]], 
                 table_name='XXXXXX', 
                 schema=os.environ['HANA_UNAME'], 
                 force=True, 
                 replace=True, 
                 object_type_as_bin=False, 
                 table_structure={
                     "RATING": "INTEGER",
                     "TITLE_WITH_TEXT": "NCLOB",
                     "TITLE": "NVARCHAR(3000)",
                     "TEXT": "NVARCHAR(5000)",
                     "SMALL_IMAGES": "NVARCHAR(2000)",
                     "LARGE_IMAGES": "NVARCHAR(2000)",
                     "ASIN": "NVARCHAR(150)",
                     "PARENT_ASIN": "NVARCHAR(150)",
                     "USER_ID": "NVARCHAR(150)",
                     "TIMESTAMP": "TIMESTAMP",
                     "HELPFUL_VOTE": "INTEGER",
                     "VERIFIED_PURCHASE": "INTEGER",
                 }, 
                 drop_exist_tab=True, 
                 allow_bigint=False, 
                 geo_cols: list = None, srid: int = 4326, primary_key: str = None, not_nulls: list = None, 
                 chunk_size=50000, disable_progressbar=False, upsert=False, append=False
             )
         else:
             dataframe.create_dataframe_from_pandas(
                 cc, chunk, 
                 table_name='XXXXXX', 
                 schema=os.environ['HANA_UNAME'], 
                 force=False, 
                 replace=False, 
                 object_type_as_bin=False, 
                 table_structure={
                     "RATING": "INTEGER",
                     "TITLE_WITH_TEXT": "NCLOB",
                     "TITLE": "NVARCHAR(3000)",
                     "TEXT": "NVARCHAR(5000)",
                     "SMALL_IMAGES": "NVARCHAR(2000)",
                     "LARGE_IMAGES": "NVARCHAR(2000)",
                     "ASIN": "NVARCHAR(150)",
                     "PARENT_ASIN": "NVARCHAR(150)",
                     "USER_ID": "NVARCHAR(150)",
                     "TIMESTAMP": "TIMESTAMP",
                     "HELPFUL_VOTE": "INTEGER",
                     "VERIFIED_PURCHASE": "INTEGER",
                 }, 
                 drop_exist_tab=False, 
                 allow_bigint=False, 
                 geo_cols: list = None, srid: int = 4326, primary_key: str = None, not_nulls: list = None, 
                 chunk_size=50000, disable_progressbar=False, upsert=False, append=True
             )
        if i == 50: break

As it is very often the case where useful information as per the customer’s sentiment is split among the title and the main context of the review, to leverage the information of both, we combined them into a single column as described below. Furthermore, the product name has been added. (sample code on the above python code).

Final Dataset

At this point we are ready to proceed with our metadata enrichment. For the needs of this study, we decided to use the Metadata Schema outlined below. We can extend our Metadata Schema upon on our needs , for example, we can add the Sentiment of the reviewer regarding the Supplier and more.

        'SENTIMENT': {
            'type': 'string',
            "description": "The sentiment of the user's review for the product they used",
            "enum": ["super positive", "positive", "neutral", "negative", "super negative"]
        },
        'PRODUCT': {
            'type': 'string',
            "description": "The product for which the review is about",
         },
        "ISSUE_TYPE": {
            "type": "string",
            "description": "A short description up to 3 words that outline the issue the reviewer is describing",
        },
        "PRICING": {
            "type": "string",
            "description": "How the user feels about the pricing of the product",
            "enum": ["Value for money", "Fair", "Overpriced"]
        },
        "SHIPPING_ISSUE": {
            "type": "string",
            "description": "A tag representing whether the reviewer was happy with delivery and shipping services or not",
            "enum": ["Overly Satisfied", "Satisfied", "Ok", "Not Satisfied", "Not Delivered"]
        }

With such a metadata schema, and for every review available in our Dataset, we can obtain useful Additional Information regarding :

User's (overall) Review ("super positive", "positive", "neutral", "negative", "super negative")
Product (The product for which the review is about)
Issue Type (A short description up to 3 words that outline the issue the reviewer is describing)
Pricing Sentiment ("Value for money", "Fair", "Overpriced")
Shipping Issue ("Overly Satisfied", "Satisfied", "Ok", "Not Satisfied", "Not Delivered")

For example,

With this information now being available and for any new customer inquiry we may receive, we can use the similarity retrieval capabilities of SAP HANA CLOUD Vector Engine to provide improved recommendations based our previous customers reviews for similar products.

Example

Since, we have completed all the previous steps, we are ready to navigate through the Data. Let's assume that we would like to see which products are good gifts for women. Furthermore, we would like to know based on their price, which ones have been reviewed as "Value for Money" or "Overpriced".

Using the below python code , we can easily run all the three above mentioned scenarios.

# ### Retrieve Documents from HANA

# Create a retriever instance of the vector store
retriever = db.as_retriever()
from langchain.prompts import PromptTemplate

prompt_template = """
You are an expert in answering questions. You are provided multiple context items that are related to the prompt you have to answer.
Use the following pieces of context to answer the question at the end.

```
{context}
```

Question: {question}
"""
PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)
chain_type_kwargs = {"prompt": PROMPT}

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory(
    memory_key="chat_history", output_key="answer", return_messages=True
)
qa_chain = ConversationalRetrievalChain.from_llm(
    llm4,
    db.as_retriever(search_kwargs={"k": 5,  "filter":{"SENTIMENT": "positive"}}),#,"PRICING": "Overpriced"}}),
    return_source_documents=True,
    memory=memory,
    verbose=False,
    combine_docs_chain_kwargs={"prompt": PROMPT},
)


question = "What is suitable gift for women?"

result = qa_chain.invoke({"question": question})
print("Answer from LLM:")
print("================")
print(result["answer"])

source_docs = result["source_documents"]
print("================")
print(f"Number of used source document chunks: {len(source_docs)}")


for doc in source_docs:
    print("-" * 80)
    print(doc.page_content)
    print(doc.metadata)

Outcome of the 1st scenario based on:

Question = "What is suitable gift for women?"
LLM Template = You are an expert in answering questions. You are provided multiple context items that are related to the prompt you have to answer. Use the following pieces of context to answer the question at the end.
Using = gpt-4-32k
Distance Strategy = Cosine (for the top 5 ones)
Metadata Filter : "SENTIMENT": "positive"

Answer from LLM:
================
Based on the provided context, suitable gifts for women could include the following options:

1. Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar: This necklace has received positive reviews and has been given as a gift multiple times, with recipients loving it.

2. Best Women Lady Retro Vintage Owl Leather Small Wallet Hasp Purse Clutch Bag: This small wallet has received a five-star review and has been described as a great gift.

3. Webat Bath Bombs Gift Set, Spa Bomb Fizzies & Ultra Lush Essential Oil: This bath bomb gift set has received a five-star review and was loved by the recipient. It is described as suitable for ladies, women, and girlfriends.

Overall, jewelry, wallets, and bath bomb gift sets are suitable gift options for women based on the provided context.
================
Number of used source document chunks: 5
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Five Stars . A gift given by request.
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': '', 'PRICING': '', 'SHIPPING_ISSUE': '', 'RID': 495, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'A gift given by request.', 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AFRFALZIA47O27PD4DKAIEDXIHAQ', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . A gift given by request.', 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Best Women Lady Retro Vintage Owl Leather Small Wallet Hasp Purse Clutch Bag and Review : Five Stars . great gift
{'SENTIMENT': 'positive', 'PRODUCT': 'Best Women Lady Retro Vintage Owl Leather Small Wallet Hasp Purse Clutch Bag', 'ISSUE_TYPE': '', 'PRICING': '', 'SHIPPING_ISSUE': '', 'RID': 12038, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'great gift', 'ASIN': 'B01J9A0LE6', 'PARENT_ASIN': 'B01J9A0LE6', 'USER_ID': 'AEDAAVKRWCWUKLJK7VQUOOGPWC5Q', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . great gift', 'TITLE_PRD': 'Best Women Lady Retro Vintage Owl Leather Small Wallet Hasp Purse Clutch Bag'}
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Necklace . Bought this for a gift and she loved it.
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': 'None', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': 'Satisfied', 'RID': 2270, 'RATING': 3, 'TITLE': 'Necklace', 'TEXT': 'Bought this for a gift and she loved it.', 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AGBMHZT2QAQX5NPQVY3K4373QL3Q', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Necklace . Bought this for a gift and she loved it.', 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Nice Gift . Gave as a gift.  Shipped to my home instead of recipient's so I had to ship it myself but quality is nice for the price!
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': 'Shipping', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': 'Not Satisfied', 'RID': 403, 'RATING': 4, 'TITLE': 'Nice Gift', 'TEXT': "Gave as a gift.  Shipped to my home instead of recipient's so I had to ship it myself but quality is nice for the price!", 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AHB5MSCRBLE7Q2W32DSRUO63TKGQ', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': "Nice Gift . Gave as a gift.  Shipped to my home instead of recipient's so I had to ship it myself but quality is nice for the price!", 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Webat Bath Bombs Gift Set, Spa Bomb Fizzies & Ultra Lush Essential Oil - Dry Skin Moisturize, Fit for Bubble & Spa Bath, Handmade Birthday Gift Ideal for Lady, Women, Girlfriend, Men and Review : Five Stars . She loved them!
{'SENTIMENT': 'positive', 'PRODUCT': 'Webat Bath Bombs Gift Set', 'ISSUE_TYPE': '', 'PRICING': '', 'SHIPPING_ISSUE': '', 'RID': 6702, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'She loved them!', 'ASIN': 'B01G45HX18', 'PARENT_ASIN': 'B01G45HX18', 'USER_ID': 'AFJY7SJ5EYKD74VHDC2Z3SWWZS6A', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . She loved them!', 'TITLE_PRD': 'Webat Bath Bombs Gift Set, Spa Bomb Fizzies & Ultra Lush Essential Oil - Dry Skin Moisturize, Fit for Bubble & Spa Bath, Handmade Birthday Gift Ideal for Lady, Women, Girlfriend, Men'}

Outcome of the 2ond scenario based on:

Question = "What is suitable gift for women?"
LLM Template = You are an expert in answering questions. You are provided multiple context items that are related to the prompt you have to answer. Use the following pieces of context to answer the question at the end.
Using = gpt-4-32k
Distance Strategy = Cosine (for the top 5 ones)
Metadata Filter : "SENTIMENT": "positive" , "PRICING": "Value for money"

Answer from LLM:
================
Based on the provided context, suitable gifts for women could include the "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar" or the "Webat Bath Bombs Gift Set, Spa Bomb Fizzies & Ultra Lush Essential Oil." These items have received positive reviews and are mentioned as being suitable gifts for women.
================
Number of used source document chunks: 5
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Necklace . Bought this for a gift and she loved it.
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': 'None', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': 'Satisfied', 'RID': 2270, 'RATING': 3, 'TITLE': 'Necklace', 'TEXT': 'Bought this for a gift and she loved it.', 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AGBMHZT2QAQX5NPQVY3K4373QL3Q', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Necklace . Bought this for a gift and she loved it.', 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Nice Gift . Gave as a gift.  Shipped to my home instead of recipient's so I had to ship it myself but quality is nice for the price!
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': 'Shipping', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': 'Not Satisfied', 'RID': 403, 'RATING': 4, 'TITLE': 'Nice Gift', 'TEXT': "Gave as a gift.  Shipped to my home instead of recipient's so I had to ship it myself but quality is nice for the price!", 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AHB5MSCRBLE7Q2W32DSRUO63TKGQ', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': "Nice Gift . Gave as a gift.  Shipped to my home instead of recipient's so I had to ship it myself but quality is nice for the price!", 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Five Stars . Gift
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': 'None', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': 'Satisfied', 'RID': 570, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'Gift', 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AHWSZ25HCWERF3KIHXNGIRJ6GQ4Q', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . Gift', 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar and Review : Four Stars . Gift
{'SENTIMENT': 'positive', 'PRODUCT': "Harlorki Women's Bohemian Jewelry Statement Necklaces", 'ISSUE_TYPE': 'Gift', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': 'Satisfied', 'RID': 636, 'RATING': 4, 'TITLE': 'Four Stars', 'TEXT': 'Gift', 'ASIN': 'B0107QYW14', 'PARENT_ASIN': 'B0107QYW14', 'USER_ID': 'AETCKBFGU6KOBXRVNDYSJ7C77NXQ', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Four Stars . Gift', 'TITLE_PRD': "Harlorki Women's Bohemian Jewelry Statement Necklaces Women Rhinestone Gem Pendant Collar"}
--------------------------------------------------------------------------------
Product : Webat Bath Bombs Gift Set, Spa Bomb Fizzies & Ultra Lush Essential Oil - Dry Skin Moisturize, Fit for Bubble & Spa Bath, Handmade Birthday Gift Ideal for Lady, Women, Girlfriend, Men and Review : Five Stars . My girlfriend really loved them. Also very in-expensive for how much they cost at most stores.
{'SENTIMENT': 'positive', 'PRODUCT': 'Webat Bath Bombs Gift Set', 'ISSUE_TYPE': '', 'PRICING': 'Value for money', 'SHIPPING_ISSUE': '', 'RID': 6518, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'My girlfriend really loved them. Also very in-expensive for how much they cost at most stores.', 'ASIN': 'B01G45HX18', 'PARENT_ASIN': 'B01G45HX18', 'USER_ID': 'AFHWPUQBBAWKC5H3VFSE2ABKXT2A', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . My girlfriend really loved them. Also very in-expensive for how much they cost at most stores.', 'TITLE_PRD': 'Webat Bath Bombs Gift Set, Spa Bomb Fizzies & Ultra Lush Essential Oil - Dry Skin Moisturize, Fit for Bubble & Spa Bath, Handmade Birthday Gift Ideal for Lady, Women, Girlfriend, Men'}

Outcome of the 3rd scenario based on:

Question = "What is suitable gift for women?"
LLM Template = You are an expert in answering questions. You are provided multiple context items that are related to the prompt you have to answer. Use the following pieces of context to answer the question at the end.
Using = gpt-4-32k
Distance Strategy = Cosine (for the top 5 ones)
Metadata Filter : "SENTIMENT": "positive" , "PRICING": "Overpriced"

Answer from LLM:
================
Based on the provided context, a suitable gift for women could be the Schöne Bath Bombs, Naughty (Pack of 6). The review states that it is a fun product, although a little expensive for regular use.
================
Number of used source document chunks: 5
--------------------------------------------------------------------------------
Product : Schöne Bath Bombs, Naughty (Pack of 6) and Review : Five Stars . LOVE. A little expensive to be purchased for regular use. For me at least. But so much fun!
{'SENTIMENT': 'positive', 'PRODUCT': 'Schöne Bath Bombs, Naughty (Pack of 6)', 'ISSUE_TYPE': 'expensive', 'PRICING': 'Overpriced', 'SHIPPING_ISSUE': 'Ok', 'RID': 8693, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'LOVE. A little expensive to be purchased for regular use. For me at least. But so much fun!', 'ASIN': 'B010TQC4R6', 'PARENT_ASIN': 'B010TQC4R6', 'USER_ID': 'AFHDOY444YBLFP6XMK5YRQI2L23A', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . LOVE. A little expensive to be purchased for regular use. For me at least. But so much fun!', 'TITLE_PRD': 'Schöne Bath Bombs, Naughty (Pack of 6)'}
--------------------------------------------------------------------------------
Product : Braun Silk Epil Female Epilator Se7921spa 1 Count and Review : Five Stars . It's a great item! Although it's quite pricey, it does the job! Highly recommended!
{'SENTIMENT': 'positive', 'PRODUCT': 'Braun Silk Epil Female Epilator Se7921spa 1 Count', 'ISSUE_TYPE': '', 'PRICING': 'Overpriced', 'SHIPPING_ISSUE': 'Satisfied', 'RID': 4446, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': "It's a great item! Although it's quite pricey, it does the job! Highly recommended!", 'ASIN': 'B00AX3YNPC', 'PARENT_ASIN': 'B00AX3YNPC', 'USER_ID': 'AE3IYBBN6KHZA5DRZUUVLER5DFMQ', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': "Five Stars . It's a great item! Although it's quite pricey, it does the job! Highly recommended!", 'TITLE_PRD': 'Braun Silk Epil Female Epilator Se7921spa 1 Count'}
--------------------------------------------------------------------------------
Product : Fleece Ear Warmers Headband for Men & Women, Thermal Polar Ear Muffs Warmers Keep You Warm and Cozy for Daily Wear, Sports, Running, Skiing and More and Review : Five Stars . Love this product.....unfortunately, I can't afford it again.
{'SENTIMENT': 'positive', 'PRODUCT': 'Fleece Ear Warmers Headband', 'ISSUE_TYPE': 'Affordability', 'PRICING': 'Overpriced', 'SHIPPING_ISSUE': 'Satisfied', 'RID': 3341, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': "Love this product.....unfortunately, I can't afford it again.", 'ASIN': 'B00DT4757A', 'PARENT_ASIN': 'B00DT4757A', 'USER_ID': 'AE22XHMBOBJBXUFCTNYLFMD4UKMA', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': "Five Stars . Love this product.....unfortunately, I can't afford it again.", 'TITLE_PRD': 'Fleece Ear Warmers Headband for Men & Women, Thermal Polar Ear Muffs Warmers Keep You Warm and Cozy for Daily Wear, Sports, Running, Skiing and More'}
--------------------------------------------------------------------------------
Product : Vivo Per Lei Shea Body Butter Lotion, Leaves Skin Softer to Touch, Devotion (Pack of 2) and Review : Five Stars . Excellent, but over priced. Can get these for a lot less.
{'SENTIMENT': 'positive', 'PRODUCT': 'Vivo Per Lei Shea Body Butter Lotion', 'ISSUE_TYPE': 'Overpriced', 'PRICING': 'Overpriced', 'SHIPPING_ISSUE': 'Ok', 'RID': 8921, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'Excellent, but over priced. Can get these for a lot less.', 'ASIN': 'B00J14AOUW', 'PARENT_ASIN': 'B00J14AOUW', 'USER_ID': 'AH563RYOFJ6THIFELPD5Z25Z3WUQ', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . Excellent, but over priced. Can get these for a lot less.', 'TITLE_PRD': 'Vivo Per Lei Shea Body Butter Lotion, Leaves Skin Softer to Touch, Devotion (Pack of 2)'}
--------------------------------------------------------------------------------
Product : Cosmetics Brushes, 10-Piece Professional Makeup Brush Set with Gift Box (White) and Review : Five Stars . good quality and service  But why the price increase?  I bought it at a great price previously
{'SENTIMENT': 'positive', 'PRODUCT': 'Cosmetics Brushes', 'ISSUE_TYPE': 'price increase', 'PRICING': 'Overpriced', 'SHIPPING_ISSUE': 'Ok', 'RID': 14836, 'RATING': 5, 'TITLE': 'Five Stars', 'TEXT': 'good quality and service  But why the price increase?  I bought it at a great price previously', 'ASIN': 'B00KQEHD46', 'PARENT_ASIN': 'B00KQEHD46', 'USER_ID': 'AFZHRCYYM6IJHNZ75THWOEMXUQSA', 'HELPFUL_VOTE': 0, 'VERIFIED_PURCHASE': 1, 'SMALL_IMAGES': '', 'LARGE_IMAGES': '', 'TITLE_WITH_TEXT': 'Five Stars . good quality and service  But why the price increase?  I bought it at a great price previously', 'TITLE_PRD': 'Cosmetics Brushes, 10-Piece Professional Makeup Brush Set with Gift Box (White)'}

Ending, in order to highlight the power of the Metadata Enrichment on GenAi scenarios , let's assume that we want to analyze the Shipping Satisfaction of the reviewers regarding the products which are good for Beauty skin. Since we have already implement this analysis , we can easily plot Word Clouds build on top of :

Reviews of Customers satisfied with Shipping
Reviews of Customers not satisfied with Shipping

Products related to Face, Skin or Eye care tend to be linked with quick delivery. However, products that come in the form of creams or are packaged in bottles may frequently experience poorer quality on shipping services.

Conclusion

In conclusion, the Metadata Tagger proves to be incredibly beneficial when a more focused similarity search is required. This is not only limited to Product Reviews , as we shown on this blog, but also on Customer reviews such as on Utility sector and many more. Furthermore, taking advantage the power SAP Business Technology Platform , it is really easy to extend the current scenario to a more sophisticated one.

I would like to thank Dr.Ing. @Dimitrios_Lyras for all his assistance on highlighting the usage and power of SAP HANA Vector Engine.

How Metadata Enrichment can enhance SAP HANA Cloud Vector Engine on GenAI scenarios

Introduction of the end-to-end Machine Learning operations with SAP AI Core

Bring Open-Source LLMs into SAP AI Core

SAP AI Core & SAP AI Launchpad - a visual introduction (Part 1)