🚀 Steps to Perform Benchmarking Evaluation Using Generative AI Hub on BTP

[Naresh_Abaka]

Here’s a concise walkthrough for setting up and running a benchmarking evaluation using the Generative AI Hub service in SAP BTP:

1. Assign Role Collection for Generative AI Hub Access

• To access the evaluation service under Generative AI Hub, assign the role collection ailaunchpad_genai_manager to the user of your BTP landscape.

• Roll Collection in BTP

2. Set Up Object Store for Evaluation

• For the benchmarking exercise, you need an object store.
• In this example, we use AWS S3 as the object store type.

3. Create Object Store Secrets

• Define two object store secrets in AICore:
  • One for input files (e.g., test datasets, orchestration config).
  • Another for evaluation results.
  • The secret for evaluation results must be named default.
  • The input file secret can be named according to your use case.
    
  • This maps to an S3 path such as:
    s3://hcp-5e9a9c35-3dd8-493a-bb55-7980c01fd279/td4
    And can be accessed using the alias:
    ai://err-res-genai-data.

4. Connect Local System to AWS S3 Object Store

• Follow Step 9 from the official learning journey to connect your local environment to AWS S3 through BTP:
  👉SAP AI Core Data Management - Step 9

5. Upload Datasets to Object Store

• Refer to Step 10 of the same tutorial to upload datasets like test data, orchestration configs, and custom metrics from your local system to the configured AWS S3 bucket.

6. Register Artifacts in AICore

• Register all relevant artifacts (test datasets, orchestration config, custom metric files) in AICore.

• Use the root directory name eval-data to register the artifcats, the sub-directory and its files are automatically registered.

• Navigate to the Other Artifacts section in the AI Launchpad dashboard and use “Add” to register the path.

Example path:
ai://err-res-genai-data/eval-data

• ai://err-res-genai-data – Refers to your object store secret key.

• eval-data – Root directory where you will create subfolders like /runs and /testdata.

7. Create & Monitor Evaluation Run

• After registering the artifacts, create your Evaluation via the AICore interface.

• Monitor the execution workflow under the Executions tab to track progress and results.

• The benchmarking metrics selected in Step 7 are evaluated based on the input artifacts and presented as follows.

[Jannis94]

Hi @Naresh_Abaka  thanks for the tutorial.

Can you provide any samples on the contents / structure of errres_run1.json and errres_dataset1.csv? I assume these have to follow certain patterns but i cant find official docs on that.

Also what kind of framework is ultimately used to evaluate, is it a proprietary SAP framework or some open-source framework under the hood?

Thanks a lot in advance.

[Naresh_Abaka]

Hi @Jannis94,

Please refer to this git page for https://github.wdf.sap.corp/ICN-ML/prompt-evaluation/tree/readme-doc/jupyter_notebook/eval-data

the sample content, also for further information refer to github page https://github.tools.sap/MLF/ml-content-blr/blob/master/custom_eval_demo/Hands-on%20Notebooks/jupyte...

Thanks,

Naresh

[Jannis94]

@Naresh_Abaka thanks for the response, but can i access these repos as a non-SAP employee / are these public repos?