Deploying a Docker Model to SAP Generative AI Hub

[ashishsingh1987]

Introduction

Deploying a custom model as a Docker container in SAP Generative AI Hub is a
practical approach when you want to expose your own inference logic as a scalable API
endpoint. This is especially useful when your model is packaged with custom runtime
dependencies, preprocessing or postprocessing logic, or when the model is not directly
available through standard managed options.

In this article, the focus is not on broader solution architecture, but on
the implementation flow required to take a trained model artifact, package it as a
Docker image, and deploy it through SAP Generative AI Hub so that it can be
consumed through an HTTP endpoint.

Why Deploy a Docker Model to SAP Generative AI Hub?

SAP Generative AI Hub provides access to a range of generative AI capabilities and
models through a governed enterprise platform. However, organizations often have AI
models that are not available as standard models in the platform.

For example, a company may have developed its own machine learning model for
customer classification, document processing, fraud detection, recommendation,
forecasting, or a domain-specific AI use case. Such a model may have been trained
internally using proprietary data and may need to be exposed as an API for consumption
by enterprise applications.

In these scenarios, the model can be packaged together with its inference logic and
dependencies into a Docker container and deployed through SAP AI Core, which
provides the underlying execution and lifecycle management capabilities used by SAP
Generative AI Hub.

This approach allows a custom model to be operated within the same broader
enterprise AI landscape while retaining control over the model runtime, dependencies,
and inference implementation. 

A simple example
Consider an organization that has trained a custom customer-intent classification
model using historical service interactions.
The model:
• Is not available in the standard model catalog.
• Has been trained using the organization's proprietary data.
• Requires a specific Python runtime and ML libraries.
• Contains custom preprocessing and postprocessing logic.
• Needs to be consumed by multiple enterprise applications.
Instead of rebuilding the model for every consuming application, the organization can
package the model and inference code into a Docker image and expose it through a
REST API.
The resulting architecture looks conceptually like this:
Trained Model → Docker Container → SAP AI Core → Deployed Inference Endpoint →
Enterprise Applications

The same approach can be used for many other custom ML models.

Typical Use Cases

Deploying a Dockerized model is particularly useful when the organization already has a
trained model and wants to make it available as a managed inference service.
Typical scenarios include:

1. Custom classification models
For example:
• Customer intent classification
• Lead or opportunity classification
• Document classification
• Risk or fraud classification
• Product categorization

The model can be packaged with its preprocessing and inference logic and exposed
through a standard REST endpoint.

2. Domain-specific machine learning models

Organizations may have models trained specifically on their business data, such as:
• Demand forecasting
• Customer churn prediction
• Recommendation models
• Anomaly detection
• Predictive maintenance
These models may not be available through a standard generative AI model catalog but
can still be operationalized through a containerized deployment.

3. Custom NLP or document-processing models
A model such as a custom NER, OCR, document classification, or text-processing
model can be packaged together with the required libraries and inference code.

4. Open-source or internally developed models
Organizations may want to operationalize an open-source model or an internally
developed model while maintaining control over the model runtime and dependencies.

5. Models with custom inference logic

A model may require more than simply sending an input and receiving an output. For
example, the inference service may perform:
Input validation → preprocessing → model inference → postprocessing → response
generation

Packaging these components into the Docker image provides a consistent runtime for
the complete inference workflow.

When Should You Choose a Docker-Based Model Deployment?

A Docker-based deployment is a good choice when you need to bring your own model
and runtime into an enterprise-managed AI environment.
Consider this approach when:
The model is not available as a standard model
If the model you need is not available through the models provided by the platform, a
custom Docker deployment provides a way to expose your own model as an inference
service.

You have a proprietary or domain-specific model
Organizations may have models trained using proprietary datasets or models
specifically designed for their business domain.
Examples include custom classification, recommendation, forecasting, anomaly
detection, or NLP models.

The model has specific runtime dependencies
Your model may require a particular Python version, ML framework, system library, or a
combination of dependencies that are difficult to reproduce in another runtime.
Docker allows the model and its runtime dependencies to be packaged together.

You require custom inference logic
The inference service may need to perform preprocessing, validation, postprocessing,
business rules, or other custom operations around the model.
Instead of exposing only the model itself, the complete inference implementation can
be packaged into the container.

You need control over compute resources
Different models have different resource requirements. A containerized deployment
allows the serving workload to be configured according to the model's CPU, memory,
and, where applicable, accelerator requirements.
Scaling parameters can also be configured according to expected workload.

You want to operationalize an existing ML model
If you already have a trained model and an inference implementation, containerizing it
can provide a relatively straightforward path to expose the model as a managed API
rather than rebuilding the model for a different serving environment.

When is a Docker-Based Model Deployment Not Necessary?

A Docker-based deployment is not necessarily the right choice for every AI use case.
You may not need this approach when:
• The required model is already available through the supported model offerings in
SAP Generative AI Hub.
• You only need standard LLM capabilities such as text generation, summarization,
classification, or embeddings and do not have a custom model to operate.
• Your requirement can be satisfied using an existing managed AI service without
maintaining your own inference container.
• The model does not require custom dependencies or inference logic.
• The additional operational responsibility of building, maintaining, securing, and
updating a container does not provide sufficient value for the use case.

Architecture

Prerequisites
• SAP BTP tenant with access to SAP Generative AI Hub and appropriate role
collections
• A trained model artifact (for example, model.pt)
• Docker installed locally and an account on a container registry (Docker Hub,
GitHub Container Registry, GitLab Container Registry, or similar)
• A Git repository to store the application code and YAML manifest
• Basic familiarity with Python (or your chosen runtime), REST APIs, and
containerization
• Personal access tokens or credentials for your Git and container registry
accounts

What You Will Build
• A simple inference service that loads your model and exposes a /predict
endpoint over HTTP
• A Docker image pushed to your container registry
• A Git-tracked YAML manifest that declares the application image and runtime
settings
• An application entry, configuration, and deployment in SAP Generative AI Hub
• An inference endpoint URL that can be consumed by authorized applications

What this article does not cover
This article is intentionally focused on the how-to steps for deployment. It does not go
into detailed solution architecture, model selection strategy, infrastructure sizing, or
comparative analysis of deployment options. Those topics can be explored separately if
needed but are outside the scope of this walkthrough.

Now let’s dive into the steps in details:

Build and push the docker image to a container registry:
• Below is an example of the commands required to create a docker file for a
python application.

One line explanation for each command:

Command	Purpose
FROM python:3.12-slim	Uses the Python 3.12 slim image as the base for the container.
WORKDIR /app	Sets /app as the current working directory inside the container.
COPY requirements.txt .	Copies the local requirements.txt file into the container's working directory.
RUN pip install -r requirements.txt	Installs all Python dependencies listed in requirements.txt during image build.
COPY . .	Copies all application files from the current local directory into the container.
CMD ["python", "main.py"]	Runs main.py when the container starts.

A useful way to remember them is:

1. FROM → Base image
2. WORKDIR → Where to work
3. COPY → Copy files
4. RUN → Execute build-time commands
5. CMD → Execute run-time command (container startup)

• Follow the steps below to create a docker image and push it to the docker container registry. Assuming your Docker Hub username is myuser, image name is myapp, and you're in the directory containing the Dockerfile.

Run the commands in terminal:

• Build the Docker image
  • docker build -t myuser/myapp:latest .
• Login to Docker Hub
  • docker login
• Push the image to Docker Hub
  • docker push myuser/myapp:latest

Add new Docker/container registry credentials

Follow the path:

Login to SAP AI Launchpad

1. SAP AI Core Administration
2. Docker Registry Secrets
3. Click Add
4. Enter the Name for your docker registry and enter the credentials for your docker registry in below format:
   {

".dockerconfigjson":

"{\"auths\":

{\"YOUR_DOCKER_REGISTRY_URL\":

{\"username\":\"YOUR_DOCKER_USERNAME\",

\"password\":\"YOUR_DOCKER_ACCESS_TOKEN\"

}    }   }"  }

Create a Git-Tracked YAML Manifest

Create a YAML file that contains:

1. Metadata

• Service name, description, scenario ID, and version.
• Information displayed in SAP AI Launchpad and AI Core.

2. Deployment Configuration

• Defines that the workload is a KServe-based serving application.
• Configures autoscaling behavior (how the service scales up/down based on incoming requests).

3. Docker Registry Access

• Specifies the Docker registry secret (imagePullSecrets) used to authenticate and pull the Docker image.

4.  Container Definition

• Specifies the Docker image to run.
• Defines the exposed application port.
• Defines the startup command used to launch the application (e.g., FastAPI/Uvicorn).

5. Resource Scaling

• Sets minimum and maximum replicas (number of running instances).
• Controls availability and scalability of the service.

This file tells SAP AI Core:

"Deploy this Docker image as a continuously running API service, keep at least N instances running, scale when needed, and expose the application so external systems can send inference requests."

Attached document contains a Template of the YAML file. 

Link the Git Repository to SAP Generative AI Hub

Follow the below steps to link the Git repository to the Gen. AI Hub:

• Login to the Gen AI Hub
• Navigate to SAP AI Core Administration à Git Repositories
• Click on “Add”
• Maintain the
  • URL end point of the Git Repository
  • Name for this Git repository
  • User Name & Access Token

Create an Application Registry Entry

Follow the below steps to link the YAML file created in previous step to the Gen. AI Hub:

1. Login to the Gen AI Hub
2. Navigate to SAP AI Core Administration -> Applications
3. Click on “Create”
4. Maintain the
   • Application Name (Name for the new entry)
   • Repository (Select the Git Repository entry created in previous step)
   • Maintain the file path to the YAML file created in previous step

Create a Configuration for the Application

Follow the below steps to create configuration reference to the application created in previous steps:

1. Login to the Gen AI Hub
2. Click on Workspaces à Select a Resource Groups
3. Click on ML Operations --> Configurations
4. Click on “Create”
5. Maintain the
   • Configuration Name (Name to the configuration)
   • Select the Scenario (Name to the application created in previous step)
   • Select the version from the drop-down value
   • Select the Executable value from the drop-down value
6. Maintain the
   • Input Parameters (If parameters are defined)
   • Input Artifacts (If parameters are defined)

       7. Review and click on “Create”

Create a Deployment from the Configuration

Once the configuration is complete, create the deployment. Once the deployment is complete and successful, end point url is generated which can be used for calling from any application to make predictions.

YAML Template For Model Endpoint Deployment.txt