Community
Artificial Intelligence Blogs Posts
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

How can we define mandatory control points in the AI agent’s decision-making chain?

What does it actually mean for a human to be "in the loop"? 

In our first blog post of this series, we introduced the concept of human oversight in AI systems and explored the spectrum of human–AI collaboration from AI assistance to full autonomy. 

Today, we make things concrete. We look at one real SAP use case that illustrates what human-in-the-loop (HITL) oversight looks like in practice: a Dispute Resolution Agent. 

The Dispute Resolution Agent 

What It Does 

The Dispute Resolution Agent supports contract accountants working on dispute cases: think incorrect invoices, missing payments, or billing discrepancies. For each incoming case, the agent automatically: 

  • Analyzes invoice details, payment histories, and contract terms, 
  • summarizes the key issues, and proposes a set of resolution options to the contract accountant. 

The human accountant then reviews the agent's analysis, inspects the reasoning and supporting documents behind each proposed option, and selects one. From that point, execution is back with the agent – the accountant's role is the decision, not the follow-through. 

Figure 1: The Dispute Resolution Agent presents a concise summary of a dispute case as well as possible resolution options to the human accountant for review and selection.

Figure 2: The AI Analysis view shows the agent's reasoning, the actions it took, other engaged agents, and the data sources accessed.

What Makes this Use Case an Example of HITL Implementation? 

It is worth being precise about what kind of oversight is happening here. Why is this a good example of the human-in-the-loop paradigm? 

The agent operates with substantial autonomy: It analyzes cases, synthesizes evidence across multiple data sources, and generates resolution proposals without being prompted step by step. The heavy lifting is done by the AI. This goes well beyond AI assistance. 

But the oversight design goes beyond human-on-the-loop (HOTL) principles too. The workflow does not allow the agent to act automatically while humans monitor and correct after the work is already done. The loop is interrupted for every case, and the accountant must select a resolution before anything is executed. Human approval is mandatory, not conditional. 

This mandatory human decision point for every consequential action is what characterizes HITL oversight. The agent cannot reach an outcome without explicit human approval. That approval is an informed decision: the agent shares its full analysis and reasoning alongside the proposed options. 

Why the Stakes Demand Human Review 

Consider what the risk landscape would look like if proposed resolutions were applied automatically: 

Broad access scope. The agent requires access to invoice records, payment histories, contract terms, and communication threads. Broad access scopes combined with autonomous execution create meaningful risk exposure. Unintended actions – such as applying credits that were not earned – can reach further than expected. 

Multi-agent collaboration. The stakes increase further when the Dispute Resolution Agent operates alongside other specialized agents covering areas such as financial accounting and cash collection. A resolution that looks correct at the dispute stage can silently trigger a chain of downstream actions: an entry removing an outstanding balance from the books, a payment release updating a customer's credit status. Each downstream agent acts on what it receives, with no way to detect that something went wrong upstream.  

Responsibility gaps. A resolution executed without human review cannot be traced back to a deliberate human judgment. If something goes wrong, accountability becomes unclear – neither the accountant nor the organization made a conscious choice. 

Goal drift.  AI systems can change over time. An agent might gradually favor faster case closure over accurate resolution. Without a human reviewing outcomes, this drift could go undetected until the consequences show up in customer complaints. 

These risks, if unaddressed, could lead to real-world harm: Dispute resolution involves financial decisions that directly affect customers and company revenue. An incorrectly resolved dispute – whether in favor of or against the customer – can have immediate financial consequences, damage relationships, and in some cases carry legal implications. The impact of errors is high. 

Human review and approval before execution address all four risks. They keep accountability with the accountant, create a natural checkpoint for detecting drift, ensure that broad access to multiple systems is mediated by a human, and mitigate the risk of downstream error propagation. The human approval step is therefore not only a check on one decision – it actively avoids real-world harm. 

The Challenges HITL Does Not Automatically Solve 

Requiring human approval is not the same as ensuring meaningful human oversight. Three well-documented failure modes are worth understanding, and each has specific implications for the dispute resolution use case. 

Automation Bias 

Automation bias is the tendency to over-rely on automated suggestions, treating them as correct by default rather than as inputs that require evaluation. It has been observed in pilots, healthcare professionals, and software developers alike [see 1-3].

In the dispute resolution context, automation bias would look like an accountant who consistently selects the agent's top recommendation without working through the underlying analysis – blindly trusting the agent's judgment rather than exercising their own. This effect may only build over time: once the agent has been performing reliably, it becomes increasingly difficult to justify the effort of scrutinizing each recommendation. The line between reasonable trust and over-reliance is easy to cross without noticing. 

One way to push back against this is to build transparency directly into the workflow: when the agent's reasoning is clearly stated, engaging with it becomes the path of least resistance. Figure 2 shows how the dispute resolution agent puts this into practice: beyond general case information, accountants can inspect the agent's chain of thought, the steps it executed, which other agents were involved, and exactly which data sources were accessed. 

Decision Fatigue

Decision fatigue is the deterioration of decision quality after a sustained sequence of decisions. The more choices a person makes, the harder it becomes to evaluate each subsequent one carefully [see 4].

An accountant processing a high volume of dispute cases in a single sitting is exposed to this risk: the work becomes monotonous. When case after case presents the same patterns, the mind stops treating each one as a new problem. Attention narrows, nuance gets missed, and genuinely anomalous cases can slip through undetected simply because they arrive after the tenth routine one. Careful evaluation is cognitively expensive, and that resource depletes. 

Structuring workloads so that accountants handle a bounded set of cases in a session, rather than an open-ended queue, reduces this exposure. The dispute resolution agent supports this by prioritizing incoming cases – so that when attention is freshest, it is directed at the cases that matter most.  

Cognitive Overload

Cognitive overload occurs when the volume or complexity of information presented to a reviewer exceeds their processing capacity. It is related to, but distinct from, decision fatigue: fatigue is about quantity over time, overload is about complexity in the moment [see 5, 6].

For the Dispute Resolution Agent, each case involves invoice records, payment histories, contract terms, agent-performed actions, and resolution options – all potentially relevant to the decision. If this information is not structured carefully, the cognitive demand of a single case can exceed what a reviewer can meaningfully process. 

Part of the answer is in how the output is presented: rather than overwhelming the accountant with everything at once, the interface shows only what is relevant to the case at hand. Figure 2 illustrates this directly for the dispute resolution agent. Information is grouped under clear headers – Actions, Summary, Data Sources, Engaged Agents – so the accountant can orient quickly. Details that would clutter the overview, such as the full text of supporting documents, are accessible through links rather than surfaced by default. 

But careful interface design only takes you so far. It works for its intended purpose only if the reviewer has the domain expertise to interrogate what the agent produces. A structured output handed to someone unfamiliar with dispute resolution does not lead to meaningful oversight. In this example, an accountant brings real knowledge of contracts, billing, and what a fair resolution looks like. 

What This Use Case Teaches Us 

The Dispute Resolution Agent illustrates what it means to implement HITL with care: the agent handles the analysis, the human retains the decision, and the design ensures that decision can be made on informed grounds. 

But the failure modes discussed above – automation bias, decision fatigue, cognitive overload – are not theoretical. They are genuine oversight challenges. Building a HITL checkpoint into a workflow is the beginning of human oversight, not the end. What makes it meaningful is the combination of transparency, workload design, and interface structure that supports the human in exercising judgment. 

Want to dive deeper? Our next post in this series asks whether trust in human oversight is itself enough to ensure that it is meaningful – and what it actually takes to validate the performance of the human in the loop.

A Note on Transparency. Speaking of ethics principles, transparency is another one we take seriously. So, in the spirit of practicing what we preach: GenAI assisted in the creation of this blog post series. Its job? Giving feedback on structure, tone, and making suggestions for clarity and conciseness across multiple iterations. The image on top of the blog post was also modified by AI. The ideas, the arguments, and the accountability remain thoroughly human.

Further Reading  

Previous in this series: 

Next in this series: 

  • Trust Isn't Enough: Validating Human-in-the-Loop for Oversight 
  • From AI Assistance to Human-on-the-Loop: A Compliance Detection Case Study 
  • When Agents Multiply: Human Oversight in the Autonomous Enterprise 

SAP resources: 

External resources: 

Sources cited: 

[1] Romeo, G., & Conti, D. (2025). Exploring automation bias in human–AI collaboration: a review and implications for explainable AI. AI & SOCIETY, 41(1), 259–278. https://link.springer.com/article/10.1007/s00146-025-02422-7  

[2] Mosier, K. L., & Manzey, D. (2019). Humans and automated decision aids: A match made in heaven? https://depositonce.tu-berlin.de/items/b2e2d621-a3af-4daa-bf19-b4bf4f5bd8eb  

[3] Wikipedia article on automation bias: https://en.wikipedia.org/wiki/Automation_bias   

[4] Sheriff Y. Ahmed, Jamshid Pardaev (2025). Human-AI Decision Dynamics: How Risk Propensity and Trust Impact Choices Through Decision Fatigue, Conditional on AI Understanding. ISSN: 2560-6018. https://www.dmame-journal.org/index.php/dmame/article/view/1484/351   

[5] Bedard, J., Kropp, M., Hsu, M., Karaman, O. T., Hawes, J., & Kellerman, G. R. (2026). When Using AI Leads to “Brain Fry.” https://hbr.org/2026/03/when-using-ai-leads-to-brain-fry   

[6] Shalu, Verma, N., Dev, K., Bhardwaj, A. B., & Kumar, K. (2025) The Cognitive Cost of AI: How AI Anxiety and Attitudes Influence Decision Fatigue in Daily Technology Use. Annals of Neurosciences, 33(1), 73–84: https://pmc.ncbi.nlm.nih.gov/articles/PMC12367725/

Comment
Labels in this area