Community
Artificial Intelligence Blogs Posts
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Read only

Engineering AI Transparency: From Build to Operations (Part 2)

Abhishek_Pant01
Advisor
Advisor
‎2025 Dec 10 3:48 PM
1,017

Introduction 

Fifty years of aviation have taught us that safety is not just documentation; it’s a culture engineered into every step of the way. In AI, safety cannot exist without transparency. If we cannot see how decisions are made, we cannot verify compliance, detect failures, or build trust. Transparency operationalizes the principles of lawfulness, ethics, and robustness within AI (As explained in the introductory blogpost of this series by Bettina Laugwitz). 

This second part of my two-part blog post on the topic of ‘Engineering AI Transparency’ moves from planning to practice. Building on Part 1 we now explore how transparency is embedded during Realization, Productization, and Operations stage. The stages where design becomes reality and governance meet day-to-day execution. Drawing lessons from aviation’s hard-earned safety culture, this second blogpost explores how aviation engineering principles like redundancy, lifecycle risk management, and continuous monitoring can inspire transparency practices that make AI systems understandable, controllable, and truly trustworthy. 

Deep dive 

Just as aviation engineered safety into every component, AI must embed transparency into every stage of its lifecycle. Black box recorders, cockpit instructions, and collision avoidance systems were not “nice-to-have” features; they became mandatory standards. Similarly, AI transparency must move beyond aspirational slogans and become part of the product, pipeline, and operations, with clear roles for every stakeholder. Research confirms this need: transparency mechanisms such as audit trails, explainable UI patterns, and incident reporting directly correlate with user trust and regulatory readiness. 

In this post, we cover the later three stages of AI innovation lifecycle: Realization, Productization, and Operations, where transparency becomes tangible through technical artefacts, user-facing design, and governance mechanisms. For the first two stages: Ideation, and Validation, refer to Part 1. Let’s take a closer look at each of these stages and the engineering practices associated with them. 

1. Realization (Building and integrating transparency related AI components) 

1.1. AI's Black‑box recorders: Action trace & auditability 

In aviation, critical flight data is saved to be used to reconstruct events in case of an incident or monitoring. This exercise then helps in identifying further improvements needed in the system. Similarly, for AI systems, we should implement action traces for all critical transactions that happen between the different system components and users, within reason. This should in no way interfere with user privacy and should in no way lead to non-consensual monitoring (look forward to an upcoming blogpost on consent and its role in AI transparency). With this action trace and auditability, we can reconstruct what the AI did and why (up to an extent, by looking at the reasoning pattern). Hence, better ensuring a fair and fact-based approach which in turn leads to higher transparency and higher user trust. 

What does it include: To make AI decisions traceable, we need a few key steps. First, linking every major transaction with a unique ID so actions can be tracked across systems. Next, setting up logging for key metrics to capture essential data for monitoring and audits. Keeping these records for a period that matches the risk level of the use case, for e.g. longer duration for high-risk cases. Alongside this, building tools that allow authorized teams to review decision paths, including inputs, outputs, versioning and model details (if needed). Privacy is critical and to ensure that it’s important that logging is done for only information that’s necessary, getting consent for sensitive data, and restricting access based on roles. Finally, all these steps should be part of governance and design reviews, so traceability becomes a design standard, not an afterthought. 

In practice: In this product tour under ‘Testing and monitoring’ we can see how Joule Studio provides testing and monitoring functionality for deployed agents as well as visibility around execution logs for enterprise-wide governance. Likewise, SAP also offers monitoring via Langchain on SAP BTP as explained in detail in this blogpost. 

1.2. AI's Pilot Override Mechanisms: Human oversight modes (HITL/HOTL/HIC) 

While certain tasks during a flight are automated and managed by software, commonly referred to as ‘autopilot’, there are also other software-driven functions beyond autopilot. However, in all these tasks the ultimate control of the aircraft always remains with the pilot. Flyers-to-be on the other hand, when deciding which seat to choose, become the deciding authority to make the final call, or in some cases, to leave that decision to the algorithm, if they wish.  

In the context of AI systems, there should always be appropriate usage of human oversight mechanisms, across Agentic AI, GenAI, or Traditional ML applications. This, in conjunction with a clearly defined responsibility delegation between the AI system and human involved, contributes to a higher level of safety for the users. This oversight mechanism residing with a human decision maker helps prevent the assistance software in a plane to do intentional/unintentional unsafe maneuvers, this decision maker could be the aircraft pilot, a maintenance crew, air traffic controllers, cabin crew, etc. based on the decision in question. We can choose and implement Human-in-the-loop (HITL), Human-on-the-loop (HOTL), and Human-in-command (HIC) based oversight mechanisms proportionate to the associated risks. 

What does it include: Effective human oversight in AI systems requires a risk-based design that classifies use cases by impact and maps appropriate modes. HITL refers to the capability for human intervention in every decision cycle of the system. HOTL refers to the capability for human intervention during the design cycle of the system and monitoring the system’s operation. HIC refers to the capability to oversee the overall activity of the AI system (including its broader economic, societal, legal and ethical impact) and the ability to decide when and how to use the system in any situation (Ethics guidelines for trustworthy AI | Shaping Europe’s digital future). Technical implementation should include control APIs for pause or rollback, integrated approval workflows, real-time dashboards with escalation triggers, fail-safe defaults, and clear automation boundaries in UI and documentation. Governance must define oversight roles, escalation paths, and regulatory alignment (e.g., EU AI Act Article 14), while training ensures operators know when and how to intervene through guidance and simulated override scenarios. Finally, auditability is essential: logging all interventions, maintaining traceability for compliance, and enabling post-incident analysis for continuous improvement (see also section ‘Incident reporting & safety boards’ below). 

In practice: In this SAP TechEd session video, you will learn more about the role that the API Management capability of SAP Integration Suite plays in enabling, governing, and securing critical agentic integrations. Alongside, in this SAP blogpost, our colleagues explain how HITL control can be added to an agent workflow via the agent builder capability in Joule Studio. 

1.3. AI's Emergency procedures: Rollback, fallback & kill switch 

While ideally, the goal is to have no accidents at all; it is an inherent practice to be prepared for the worse which means having mitigation strategies for all levels of risk. In aviation, emergency procedures are meticulously engineered to ensure maximum safety and minimum disruption during crises. These procedures are also meticulously documented and communicated to relevant stakeholders, so they are prepared to do the needful when the system fails. These procedures are not hidden, rather they are visible, understandable, and governed (as applicable). 

Similarly, to handle the scenario where an AI system partially or completely fails, especially those embedded in critical workflows, the emergency mechanisms should function as a path of maximum safety and minimum disruption. Restarting and “hoping for the best” may work when your laptop freezes while watching a movie. In critical AI scenarios, this approach is not an option.  

These emergency mechanisms should not only exist but be visible, understandable, and governed. This is particularly important as AI failures are often non-deterministic and can propagate silently through backend workflows, unlike traditional software crashes. Also, for AI systems in critical sectors e.g. healthcare, finance, etc., we require a safety net to prevent cascading harm (Where vulnerabilities overlap and interact, further escalation points are created that can create secondary effects with greater impact than the primary event). Alongside this, regulatory frameworks like the EU AI act , mandate robust oversight and emergency controls for high-risk AI systems (Article 14).  

What does it include: Emergency mechanisms related transparency for AI systems should include layered safeguards such as documented rollback, fallback, and kill-switch mechanisms with clear triggers and escalation paths related information. These controls must be clear and understandable, providing visibility into why a fallback was activated, for e.g. whether due to threshold breaches or anomaly detection, either through logs or UI indicators (or any other such components). Comprehensive auditability is essential, with all emergency actions logged alongside timestamps, decision rationale, and authorized approvals for compliance checks. Role-based transparency should ensure that operators, administrators, and auditors receive appropriate details, for e.g. technical specifics for engineers to high-level summaries for compliance teams. Finally, governance must embed these transparency measures into pre-release checklists and continuous integration (CI) and continuous delivery (CD) pipelines, verifying that emergency controls are tested and validated before deployment. (Pre-release checklist elaborated in Part 1.) 

In practice: Here we can see how SAP offers restore point functionality in its cloud operations. Within a use case, we can see how SAP provides scalable, maintainable, and resilient test automation in Success Factors projects. These sandboxes can also be used to check the transparency functionalities related to emergency mechanisms. 

2. Productization (User experience and deployment readiness) 

2.1. AI's Flight manuals & Standard operating procedures: Product Documentation 

In aviation, there are numerous manuals (For e.g. SOP FOR FLIGHT DECK CREW MEMBERS) which are critically important and hence a mandatory part of the whole process to share information around major components that are used at different stages. These manuals answer questions like how different components function, how they interact with other components, and how they should be operated. Similar to this, for AI, there should be provisioning of diverse catalogs which should include information around APIs, Data Products, Metadata, and Data Lineage with that specific offering to ensure seamless integration, efficient usage and robust governance. This promotes literacy of stakeholders around product usage, contributes to appropriate traceability across systems, while empowering innovation and informed decision-making through streamlined, accessible data assets. The end goal of this practice should be to ensure that we have the answers to a variety of questions raised around the system's capabilities and usage practices. Sample questions that should be answered by this documentation: What is this system capable of? What are the major components within this system? How should different stakeholders use this system within the bounds of its acceptable usage? And also, information about what kind of threat modeling has been performed on the system? This should not be confused with giving everyone access to everything. The access to the respective modules of information should be role specific. 

What does it include: The documentation should clearly define its scope and provide a system overview of outlining purpose, capabilities, and limitations (see SAP AI Ethics policy chapter 7.7). It must include component catalogs such as APIs, data products, metadata schemas, and lineage, along with integration guidelines describing interactions and usage boundaries. Governance hooks like role-based access, compliance checkpoints, and audit trails are essential. Alongside this, standardization is key, hence using machine-readable formats (e.g., OpenAPI, JSON/YAML) complemented by human-readable guides and visual diagrams for architecture and data flow should be used. Embedding role-based access controls for sensitive sections, applying least privilege principles for external stakeholders. Integrating documentation updates into continuous integration (CI) and continuous delivery (CD) pipelines with automated versioning and changelogs for traceability. Hosting all manuals in a secure, searchable repository with tagging and enterprise system integration for easy discovery. Finally, ensure compliance and quality by aligning with ISO/IEC AI standards (e. g. ISO/IEC 42001:2023) and internal Responsible AI policies (e. g. SAP’s AI Ethics Policy), and include transparency and explainability checkpoints in SOPs. 

In practice: Here is a great example of documentation with clear scope and relevant details for SAP Joule Integration. These two(1, 2) pieces shed light on the importance of threat modeling and how SAP has incorporated that within its processes. 

2.2. AI Transparency's Redundant systems: Two layers of understanding 

Within aviation, redundant systems, i.e. functionally identical or comparable resources, are designed to prevent single points of failure. For example, in a three-way redundant system, an error in one component can be outvoted by the other two. These systems are rarely needed during normal operation but are critical for safety when failures occur. Similarly, AI systems should incorporate redundant mechanisms for transparency and explainability. If one layer of transparency fails or becomes inaccessible, another should provide the necessary clarity to maintain trust and compliance. Transparency isn’t static: systems evolve, user contexts shift, and failures can happen. Redundant transparency mechanisms ensure resilience, enabling stakeholders to understand and audit decisions even if one approach breaks or proves insufficient. This concept aligns with the principle of Understandability, as highlighted in this other SAP blogpost on Trustworthy AI, which emphasizes making AI decisions interpretable and meaningful for diverse stakeholders. These layers will always be specific to the AI use case and informed by user research into stakeholder needs. For more details, see the explainability blogpost.  

What does it include: Dynamic fallback for transparency related information with different layers, including primary (rich and interactive transparency components including detailed dashboards, UI components with details, etc.), secondary (simplified yet essential transparency components including static reports, summary logs, etc.) and tertiary (minimal compliance level transparency components including audit snippets, version history, etc.)(For further details reference to ISO/IEC 42001:2023 - AI management systems, ISO/IEC 23894:2023 - AI — Guidance on risk management). Again, the components in these layers would be highly dependent on the use-case needs.  

Secondly, there should be a failover logic which exists between these layers, for example if the primary system is unavailable or fails, the secondary layer automatically surfaces essential details.  Thirdly, this fallback logic should be part of the overall transparency architecture of the AI system. Lastly, there should be linked health checks and monitoring features to trigger fallback modes dynamically. 

UX Related Note: While these layers address operational and compliance needs, end-user experience should follow progressive disclosure principles, starting with simple, digestible information and revealing more detail as needed (as mentioned in our explainable AI principles here). This ensures transparency mechanisms remain usable and meaningful for diverse stakeholders. 

2.3. AI Cockpit instructions: User assistance for operational clarity 

A lot of engineering marvels happen on the production side of aviation but for a pilot, one of the most important things is the cockpit instruction manual. It needs to include information related to what kind of plane they are flying, what components are there, what the operating procedure is for all these components, and how to troubleshoot. They care less about the development aspect but more about the operations. Similarly, for the different users who are on the pilot seat at different stages, whether it is the business admin, IT admin, or other operations personas, it is important that they have a clear set of instructions around different components, their functions and how to operate them. This assistance ensures that they understand the entire operations environment without being overwhelmed by development details. 

What does it include: Comprehensive, stable documentation that explains the system’s purpose, components, and operating procedures in a way that is easy to understand and role specific. It should include clear descriptions of each component and its function, step-by-step guidance for configuration and troubleshooting, and tailored instructions for different personas, such as operational clarity and compliance status for business admins, integration and setup details for IT admins, and actionable workflows for operations users. To ensure accessibility, this information should be available in searchable online repositories, complemented by embedded help links and visual diagrams within the user interface. Finally, user assistance must be continuously updated through automated version control and integrated into CI/CD pipelines to maintain accuracy and traceability. 

In practice: SAP takes a structured, persona-centric approach to user assistance for AI systems by providing comprehensive documentation and contextual help integrated into its platforms. SAP Help Portal offers detailed guides on configuration, operations, and governance for AI services, ensuring clarity for business and IT administrators alike (https://help.sap.com). Additionally, SAP embeds user assistance directly into the UI through guided workflows, consent prompts, and contextual help links, making operational guidance accessible without leaving the application. 

2.4. Passenger safety cards: AI Notice & plain‑language explanations 

While the end user does take the role of a pilot at some point and hence the cockpit instructions sections cover some part of the end user transparency and guidance, I still want to explicitly mention the end user and impacted user transparency because there are still some aspects left to discuss. One might think that the passenger safety card in planes is outside the realm of engineering but would only be partially correct. While there are other aspects to the passenger safety cards which fall outside engineering, but a big chunk of the information provided on these cards and the other instructions provided alongside is carefully engineered to communicate with the end user which is the same as impacted users in the case of aviation(up to an extent). Similarly, in AI systems, end user and impacted user transparency play a crucial role. End users need to understand what the AI is doing, why, and what their options are, in different scenarios, without requiring technical expertise. For impacted users, those who may not directly interact with the AI but are affected by its decisions, this transparency ensures they can understand the rationale behind outcomes and have channels to contest or seek recourse when necessary. 

What does it include: AI notice indicators should clearly signal when AI is involved in a decision or recommendation using consistent visual cues (e.g., “AI-generated” badge) placed contextually near the relevant component. Alongside this, providing plain-language explanations in three levels (As mentioned in UI Explanation Patterns section): a quick summary in one sentence, a short paragraph explaining why the AI made the recommendation, and an optional deep dive for advanced users with influencing factors, and other appropriate details. This should follow a persona-centric approach, for e.g. business and IT admins need operational clarity and compliance status; end users require actionable guidance; and customer admins need oversight controls for review and override. Additionally, ensure accessibility and localization by using jargon-free language, supporting multiple languages, and complying with WCAG standards. Finally, validate clarity through continuous user research and comprehension testing to maintain trust and usability. 

In practice: This principle is already reflected in design systems like SAP Fiori’s AI Notice pattern, which integrates visual indicators and layered explanations directly into the user interface. Alongside this this blogpost shares information on SAP’s adoption of WCAG 2.2(Accessibility standards) within its UI5 transition.  

3. Operations (Monitoring, learning and continuous improvement) 

3.1. AI's Collision Avoidance System: Routing and Scenario collision detection 

Similar to the aviation traffic collision avoidance system (TCAS), which prevents mid-air conflicts, AI systems need mechanisms to avoid unsafe overlaps, or conflicting actions across workflows. This is especially critical in Agentic AI and Multi-model orchestration. It can help in avoiding failures, for example two agents acting on the same resource without coordination can cause data corruption. From an AI transparency perspective, it is crucial to explain the routing logic to necessary stakeholders, for example, why one scenario was prioritized over another. It is also important to ensure that the action taken by the AI does not have any unintended consequences as scenario overlaps can create new action possibilities within the system. The goal here should be to catch “near‑misses” early and tell users what changed and why. 

What does it include: To prevent conflicting actions, AI systems should maintain a centralized routing engine (At assistance or orchestrator level) that maps all workflows and dependencies, and applies priority rules (e.g., compliance over convenience). This engine should support dynamic rerouting or queuing when conflicts arise. Crucially, the decision logic must be made clear and transparent at all times through logged reasoning, UI indicators, and clear communication of why scenarios were delayed, blocked, or rerouted. Robust simulation and stress testing under peak loads and edge cases are essential to validate these mechanisms. Finally, audit and governance processes should cover routing decisions to ensure full traceability and compliance with organizational and regulatory standards. 

In practice: SAP AI Core provides the necessary transparency for monitoring resource usage. Customers can access their resource groups from their SAP AI Core runtime and perform actions on only the AI assets they contain, such as models, executions, and deployments. Secondly, SAP enterprise performance testing provides cloud-based testing to ensure applications perform under heavy user loads and support digital transformation. One more example is SAP extending Joule Agents beyond software into the physical world, connecting our client’s business intelligence to robotics and industrial systems. This opens a new frontier for automation, combining state of the art electronics that utilize physical AI tools, such as computer vision and collision detection, with AI agents capable of reasoning through complex goals and planning multi-step workflows. 

3.2. Incident reporting & safety boards: Post‑release evidence packs 

Learnings from our success and failures is a crucial part of making progress. In aviation, every incident, no matter how minor, is thoroughly investigated, and the findings are used to influence future innovation and safety protocols. Similarly, for AI systems, post-release transparency is critical to ensure continuous improvement and trust. This means creating structured feedback channels, collecting incident reports, and maintaining evidence packs that document what happened, why, and how it was resolved (Along with the Blackbox based audit trails mentioned in the beginning). This creates a shared transparency-based environment which helps us create more holistic solutions. 

What does it include: Post-release transparency should start with robust feedback channels that allow reporting from all relevant personas: end users, admins, engineers, and compliance officers; via in-product options like “Report Issue” buttons and anonymous submissions for sensitive concerns. Incidents must be categorized into technical failures, ethical breaches, and operational risks, with standardized templates for evidence packs capturing summaries, timelines, system snapshots, root cause analysis, and corrective actions stored in a secure repository for audit readiness. Alongside this, governance should include an AI Safety Board with cross-functional representation to review reports, approve actions, and feed learnings into Responsible AI guidelines and future standards. Continuous learning loops should integrate findings into feature retraining, design guidelines, and updating transparency prerequisite checklist, while compliance measures ensure alignment with frameworks like the EU AI Act and ISO 42001, maintaining traceability and publishing internal/external transparency reports. 

In practice: We at SAP are committed to sharing what we learn through resources like the SAP Trust Center.  Which also has features like incident reporting and speak out to collect feedback. SAP AI management system is already ISO 42001 certified. 

Closing 

While we do understand that the engineering aspect of AI Transparency and Explainability is an ever-evolving space and it is impossible to cover everything within a single piece. But at the same time, we believe we must start somewhere. This blogpost is an attempt to share the major engineering aspects associated with the topic. At SAP, we have a lot of these practices already in place and are actively working on the rest. As this area develops with new research and advances in techniques and measures, we will try to keep posting updates. Aviation safety became non-negotiable because every stakeholder embraced their role. AI transparency must follow the same path: engineering practices embedded in governance, supported by culture, and reinforced by accountability. When transparency is treated as a shared responsibility, trust isn’t just promised-it’s engineered. 

Abhishek_Pant01_0-1765379080319.png

 

 

Let us keep sharing how AI transparency helps keep_the_blue_side_up. 

Stay tuned for our upcoming blog posts on AI transparency where we take a look at its different aspects in detail.

 

References and further reading

Responsible AI

SAP Capabilities