Leveraging Image Embedding and HANA Vector Engine for Reverse Product Search

[HarukiSuzawa]

Introduction

HANA Cloud Vector Engine significantly accelerates the development of Retrieval-Augmented Generation (RAG) applications. Traditionally, RAG solutions have focused on leveraging text data from internal documents. However, with the rise of multimodal AI, as seen with GPT-4o, there is a growing trend towards accepting various input types such as images, audio, and video. To maximize the power of Vector Engine, integrating image similarity search is essential for enhancing its capabilities.

Reference: HANA Database Vector Engine Guide

Use Cases

Imagine a potential customer visiting a friend’s house and discovering an amazing chair. However, his friend can’t remember where he bought it. This is where Reverse Product Search comes into play. The customer takes a picture of the chair and finds that your company sells the exact same model. Reverse Product Search is particularly useful when customers cannot identify a product by name or barcode. It’s also ideal for finding repair parts or similar items.

Image Similarity Search with HANA Vector Engine

Embedding Product Images

The first step is to convert images into num erical data so that a computer can measure the similarity between them. In this prototype, we used OpenAI's CLIP, a zero-shot model that links images and text for cross-modal understanding without task-specific training.

from langchain_experimental.open_clip import OpenCLIPEmbeddings

clip_embd = OpenCLIPEmbeddings(model_name="ViT-g-14", checkpoint="laion2b_s34b_b88k")

This code embeds the product images and stores them in HANA Cloud.

Product images are preloaded into image_storage using the Product Master API.

def process_images(clip_embd, hana_conn, image_storage):
    
    cursor = hana_conn.cursor()
    images = image_storage.get_images()

    csv_rows = []
    for i, img in enumerate(images):
        model = 'OpenCLIP'
        embedding =  clip_embd.embed_image([BytesIO(img["data"])])
        csv_rows.append([i,img["name"],model,str(embedding[0])])

    sql = '''
      INSERT INTO EMBEDDING_CONTENT ("id","content","model","EMBEDDING")
      VALUES (?,?,?,TO_REAL_VECTOR(?))
    '''
    
    try:
        cursor.executemany(sql, csv_rows) 
    except Exception as e:
        hana_conn.rollback()
        print("An error occurred:", e)        
    finally:
        cursor.close()

Retrieving Similar Images

Once the product images are embedded and stored in HANA Cloud, you can retrieve images similar to a photo taken by a smartphone. Specifically, this code embeds the mobile_image and compares the cosine similarity between the mobile_image and each image in HANA Cloud. Finally, it retrieves and ranks the top 5 images based on their similarity scores.

def process_mobile_image(clip_embd, hana_conn, mobile_image):
    
    cursor = hana_conn.cursor() 
    embedding =  clip_embd.embed_image([BytesIO(mobile_image)])
    
    sql = '''
      SELECT TOP 5 "id"
      FROM "EMBEDDING_CONTENT"
      ORDER BY COSINE_SIMILARITY("EMBEDDING", TO_REAL_VECTOR(?)) DESC
    '''
    
    try:
        cursor.execute(sql, (str(embedding[0]),))
        retrieved_products = cursor.fetchall()
    except Exception as e:
        hana_conn.rollback()
        print("An error occurred:", e)        
    finally: 
        cursor.close()
        return retrieved_products

Reverse Product Search with BTP SDK for iOS

Interacting with Product Master API

Product images are managed as attachments of product master data so that they can be easily accessed by application.
To download attachments, you have to use Get All Originals operation to retrieve the metadata of the attachment.
Reference: Get All Originals

GET <host>/sap/opu/odata/sap/API_CV_ATTACHMENT_SRV/GetAllOriginals?LinkedSAPObjectKey='TestEKKO'&BusinessObjectTypeName='EKKO_RFQ'

Using the metadata retrieved, you can download an attachment from the product master data.
Reference: Download Attachment Content

GET<host>/sap/opu/odata/sap/API_CV_ATTACHMENT_SRV/AttachmentContentSet(DocumentInfoRecordDocType='SL1',DocumentInfoRecordDocNumber='10000002076',DocumentInfoRecordDocPart='000',DocumentInfoRecordDocVersion='00',LogicalDocument='1C98EC1818551EDA998508E3A05E0BCE',ArchiveDocumentID='1C98EC1818551EDA998508E3A05E4BCE',LinkedSAPObjectKey='TESTEKKO',BusinessObjectTypeName='EKKO_RFQ')/$value

BTP SDK for iOS

To build an end-to-end solution, I have developed an iOS application using the BTP SDK for iOS. This application captures images, sends them to a server-side application, and displays a ranking of similar products.

Conclusion

As we have explored, integrating Image Similarity Search unveils new potential for HANA Vector Engine. Additionally, seamless integration with SAP technologies such as S/4HANA Product Master and BTP SDK for iOS enables the efficient development of sophisticated and innovative enterprise applications.

References

Enhancing S/4HANA with SAP HANA Cloud Vector Store and GenAI

First steps using the Hana Vector Engine with SAP GEN AI

Mastering SAP HANA Cloud Vector Engine for RAG-driven LLM Applications: An Insightful Guide

[SARK]

Great blog! I really appreciate how you showed the practical use of HANA vector stores for image storage and retrieval in the context of Reverse Product Search. It's impressive how HANA can efficiently handle and retrieve images using vector representation, making it easier for customers to find exactly what they’re looking for just by taking a photo. Your explanation of this technology in a real-world example, like finding a chair someone admires at a friend’s house, really brought the concept home. It's clear that this approach not only enhances search accuracy but also adds real value to the customer experience. Thanks for breaking it down so well!