- SAP Managed Tags
Introduction
This blog is publishing my internal notes on SAP's AI infrastructure. Its September 2024, this is moving really fast.
SAP has a vast and powerful suite of services and products to manage the AI data infrastructure layer that will bring custom AI applications to life. No mater how much we initially plan on using SAP's brand new technology, the gravity of SAP's data require a curation of the transactional elements to feed a significant number of algorithms.
The demand for deriving insights from various sources of information has been around for a long time, but the rise of GenAI has brought about a new set of applications that require a substantial amount of this vital resource. To ensure optimal performance during both the training and inference stages, it is crucial to have access to top-notch data, and organizations are actively seeking methods to obtain it. Moreover, the nature of the data being collected is undergoing a transformation, expanding beyond traditional formats like text and tables to encompass multimedia elements such as videos, images, and audio recordings.
Across the data ecosystem, we’re witnessing innovation in areas like unstructured data extraction and pipelining, retrieval-augmented generation (RAG), data curation, storage, and AI memory.
This diagram aims to order down the AI data infrastructure landscape, but first, it's crucial to establish a basic understanding of what I believe it's SAP's current data infrastructure landscape.
SAP AI Data Infrastructure Landscape
When designing this diagram, my goal was to simplify the representation of how data flows through the AI value chain, from Ingestion to Serving.
The data infrastructure value chain can be broken into four key components:
1. Sources
2. Ingestion & Transformation
3. Training and Inference
4. Data Services
This visualization serves as a conceptual framework rather than a comprehensive list. Here’s a brief explanation of each segment:
Sources
Data sources and types can differ depending on the use case. Traditionally, a company’s business data is housed within systems like S/4HANA or ECC, while transactional data is stored in databases such as HANA or Oracle. Real-time data is often pulled from other sources, such as sensors, manufacturing systems, and healthcare applications, which I broadly categorize as “real-time” data.
Data Ingestion and Transformation
Once data sources are selected, the next step is for companies to ingest, transform, and transport the data to a destination where it can be effectively utilized. This is also not new, and the primary purpose of data pipelines is straightforward: move data from its source to its destination in a format that is easy to analyze or act upon. Traditionally, in data engineering, this is done through ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform). In machine learning, where much of the data is structured, this process is known as feature engineering or feature pipelines. However, with the rise of Generative AI, the focus has shifted to extracting, parsing, and preparing unstructured data, collectively referred to here as data pipelines. While data pipelines have been around for decades, the main change today is the diversity and sheer volume of data that needs to be processed, its not about text, its about images, sound and video.
Data pipelines typically fall into two main types: batch (where data is extracted and loaded at set intervals) and streaming (where data is processed as soon as it becomes available). A new type of pipeline has also emerged for handling unstructured data, offering end-to-end workflows that move this data to storage.
Tools in BTP like CPI or Mesh serve as orchestrators, managing the scheduling, execution, and organization of these workflows.
For machine learning training, data must be filtered and labeled. Labeling provides the necessary context so that models can learn patterns from the data. Supervised learning, in particular, requires well-labeled datasets to ensure the model learns what is correct and what is incorrect.
Model Training
Although SAP still provides limited capabilities in Model Training, AI algorithms generally rely on three main types of training: supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the model is trained on labeled data, learning to predict outcomes that match the given labels. In unsupervised learning, the model is fed large datasets and learns to identify patterns and relationships without any explicit labels.
For large language models (LLMs), "pre-training" typically involves unsupervised learning, allowing the model to identify patterns within the data. Afterward, the model is further refined using supervised learning to improve its performance. Custom machine learning models are usually trained using supervised learning as well.
Next, models undergo reinforcement learning, specifically Reinforcement Learning from Human Feedback (RLHF). In this phase, the model generates outputs and receives feedback from humans on how to improve its performance, allowing for continuous refinement.
Throughout the training process, models are evaluated for how accurately they represent a given task or situation. Key metrics include accuracy, precision, loss minimization, and checks for overfitting or underfitting, depending on the specific use case of the model.
Finally, before deployment, models undergo several final steps such as security testing, governance reviews, and auditing to ensure they are safe, secure, and compliant with relevant standards.
Model Inference and RAG
In LLM inference, when the model receives a prompt, it first tokenizes and vectorizes the input data (a process known as "prefilling"). The data is then processed through the model to generate an output for the user, a step referred to as "decoding."
When personalization is required, the process becomes more complex. For instance, a company might store data in a vector database and link it to an LLM via a customization platform. When a user provides a prompt, data from the vector database is used to create a tailored response through the LLM. This setup can also be employed by AI agents to incorporate contextual knowledge of a company or user and act on their behalf.
Throughout this process, it’s essential to track and manage the data to ensure the security, quality, and compliance of the model—this is where data services come into play.
RAG is a workflow that enhances LLMs by integrating external, custom data. In RAG, data is indexed and queried, allowing the LLM to access the most relevant information alongside the prompt. This setup ensures that LLMs can produce up-to-date, accurate responses by referencing curated data sources. RAG improves reliability, enhances factual accuracy, and enables AI models to provide citations, building trust with users.
SAP Reference Architecture for Artificial Intelligence Technology
The artificial intelligence technologies that are used in SAP systems to implement corresponding business applications.
I identified two types of technology provisioning in the context of ERP systems. There are artificial intelligence technologies that are embedded into the database system and application server of ERP systems. For scalability reasons, artificial intelligence infrastructure (e.g., GPU computing) is also provided side by side to the ERP platform.
There are two variants of artificial intelligence technologies provided in context of SAP ERP. Basic libraries and runtimes are available on the ERP platform for implementing simple scenarios. In addition, AI technology platforms like AWS AI, Azure AI, Google AI, IBM Watson, Alibaba AI, or SAP BTP are supported side by side to the ERP platform for complex scenarios.
These AI technology platforms typically share a similar structure, offering infrastructure for training, inference, data storage, GPU hardware, operations, and monitoring. They also support data science environments and generic services like image recognition and text translation. My aim is to give an overview of some of SAP’s most significant artificial intelligence technologies. While this document does not cover all the features and functionalities of these technologies, it does offer guidance on their usage and recommendations on which technology to use in specific situations, as some have overlapping capabilities. SAP provides a range of artificial intelligence technologies to its partners, clients, and internal stakeholders for their own projects. I will discuss SAP HANA, SAP Data Intelligence, SAP AI Core, SAP Generative AI Hub, SAP Datasphere and SAP Analytics Cloud and conclude with SAP AI Business Services and SAP AI Launchpad.
Based on the ERP reference artificial intelligence technology, we can understand SAP HANA as embedded technology of the ERP platform, while all the other listed technologies are provided side by side on the AI technology platform of SAP. I dont provide a distinction between an SAP product, an SAP tool or an SAP service.
SAP HANA’s key feature is its multipurpose database, which allows users to store, process, train, and serve all their data and artificial intelligence processes in memory and in real time. As all customer-initiated actions and operations are executed immediately within SAP HANA’s in-memory database, there is no need to transfer data to another system for processing. The specialized machine learning (ML) libraries "Automated Predictive Library" (APL) and "Predictive Analytics Library" (PAL) built into SAP HANA applications support a wide range of artificial intelligence use cases. For data scientists’ convenience, all training methods offer a native scripting interface (SQLScript), which can be used directly or wrapped in Python and R libraries.
When it comes to complex orchestration situations, data categorization, and data quality procedures, SAP Data Intelligence excels in these areas. It can seamlessly integrate unstructured, streaming, or cloud application data in various formats scattered across the organization and write it to the desired endpoint. With connections to R, Python, APL, and PAL libraries, SAP Data Intelligence provides a unified graphical design interface for both data ingestion and transformation. SAP Data Intelligence is suggested for situations where artificial intelligence use cases involve multiple diverse data sources that need to be merged and managed in SAP HANA, with an R Server, or directly in a Python environment. Data Intelligence and Datasphere also supports data orchestration to external artificial intelligence environments. When advanced hardware resources like GPUs or intricate orchestration of workflow steps are required, SAP AI Core is the recommended solution for managing and controlling training and serving workflows in a scalable AI runtime. It is designed for AI engineers with strong coding skills and a need for flexibility. SAP AI Core aims to operate scalable, cost-effective, and customizable artificial intelligence models while maintaining privacy and compliance. It ensures the high level of scalability for every artificial intelligence scenario through auto-scaling, scale-to-zero, multi-model serving, and a broad array of resource types, including GPU support.
For analytical and business users, SAP Analytics Cloud offers built-in predictive capabilities with a simple user interface. Its prediction engine is constructed using the APL library from SAP HANA. Live datasets can be created on top of SAP HANA on-premises systems, and data can be collected from multiple source systems. Forecasts made using SAP Analytics Cloud’s predictive capabilities are typically consumed through SAP Analytics Cloud stories. SAP AI Business Services provide strategic machine learning capabilities that enhance customer experiences by automating and optimizing business operations. These services and applications are available as reusable, generic offerings that can be immediately utilized. Most of these services use SAP AI Core as the underlying artificial intelligence environment. SAP AI Launchpad and Generative AI Hub serves as a centralized tool for managing the life cycle of artificial intelligence models, deployments, and other operations-related information across all deployment scenarios and landscapes. It also allows users to manage supporting AI runtimes like SAP AI Core, SAP HANA, and SAP Data Intelligence. SAP AI Launchpad becomes the standardized user interface for managing and operating any artificial intelligence use cases provided by SAP or custom-developed, due to the centrally regulated AI API for life cycle management. Based on the AI API abstraction, third-party artificial intelligence models can also be utilized for implementing artificial intelligence applications.
Data Preparation in AI activities
Data preparation and quality assessment to ensure the collected data is fitting for its intended purpose. While data science cannot glean insights from poor-quality data, it can certainly do so when applied correctly to a dataset of adequate quality. For this, SAP HANA offers tools for AI data preparation, including SQL scripts, Predictive Analysis Library (PAL), Automated Predictive Library (APL), and machine learning clients for Python and R. PAL provides advanced pre-processing tools, while APL automates handling of data issues. SAP Data Intelligence integrates data across various formats, ensuring reliable, scalable data preparation for AI projects. SAP AI Core manages complex workflows on Kubernetes for scalable AI solutions, especially when GPU support and advanced orchestration are required. SAP Datasphere and SAP BTP technologies like Generative AI Hub enhance data preparation by unifying data from multiple sources and enabling AI-driven data insights across SAP system.
Data Modeling
Data modelling, involves each step and technique to apply to the data in order to achieve the specified goal and executing these steps.
SAP HANA Cloud multi-model database enables artificial intelligence scenarios to take advantage of its diverse features. Both Calculation View and Smart Data Integration Flowgraphs play a vital role in Data modeling, offering the flexibility of pure SQL and SQL Script data manipulation.
Datasphere is the other star product. Datasphere powerful data modeling features cloud-based data integrated from diverse sources, both structured and unstructured. Data engineers can combine, clean and prepare data using graphical views, tables, relationship models, SQL views, analytical models, data flows, and more. Then business users create semantic models on top of data objects.
Data Evaluation
The nature of SAP data will reside on HANA or HANA Cloud, so due to the fact that SAP HANA’s integrated artificial intelligence is based on industry norms, the applied algorithms deliver all the conventional metrics needed to assess the performance of the artificial intelligence model, as one would expect from any library. These metrics are typically generated by default during training, cross-validation, or score function runs and are included in the standard output.
The APL library also includes two exclusive metrics for model performance, Predictive Power and Prediction Confidence, which are intended to provide business users with a more intuitive understanding of model performance. Of course, all generated metrics can be natively accessed through the machine learning clients for Python and R.
For non-SAP generated data, SAP Datasphere offers powerful features for evaluation. Integrating diverse data sources, both SAP and non-SAP, structured and unstructured, helps data engineers to combine, clean, transform and link data using graphical tools, SQL, and data flows. Then business users create semantic models on top of Data Builder objects. Advanced analytics capabilities like data mining, machine learning, and predictive analysis help discover insights from the integrated data.
Data Deployment
AI Deployment in SAP is moving really fast. SAP AI Core and Generative AI Hub offer powerful features for deploying large language models (LLMs) which is capturing a lot of attention due to its inmense possiblities.
Through BTP, we access to a wide range of LLMs from different providers, including open-source models and proxies for GPT, Google, and Amazon models. It is easy and well documented the deployment of local LLM proxies by creating a configuration specifying the model name and version. Deploying embedding models for retrieval augmented generation (RAG) use cases took blogs and webinars in the past weeks, while integration with SAP AI Launchpad provides a user-friendly interface to deploy and manage LLMs.
Conclusion
In this blogpost I present a set of notes and diagrams I have been building for my personal use over the last weeks and months. I initially separate the AI Pipeline in four steps, the Source, the Ingestion, the Training and the Service. Then I spare the deployment of AI technologies in four groups; Preparation, Modeling, Evaluation and Deployment. All the categories are grouped around mostly 6 key Products or BTP Services around it, trying to simplify the terms.
