Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Showing results for 
Search instead for 
Did you mean: 
ChristophBatke
Associate
Associate
873

Document Information Extraction - Premium

With the general availability of Document Information Extraction, premium edition (DOX Premium), users now have the power to leverage Generative AI (GenAI) for extracting entities from structured documents. This cutting-edge capability allows for processing a wide variety of document types, in almost every language. However, as with most GenAI-driven applications, the success of the extraction process heavily depends on the quality of the prompts given to the model.

This blog post explores best practices for using GenAI in document extraction, focusing on optimizing performance through effective prompt engineering, refining field descriptions, and ensuring continuous improvement. By applying these insights, you can maximize the potential of GenAI-driven document processing in your organization.

The process of iterative prompt development

When working with large language models, it's essential to understand that the first query is rarely perfect. For each field requiring extraction, we begin by writing a base description that captures essential information (such as position, format, or context) about the entity. Once we review the results, we often need to refine the input based on the initial output.

However, a word of caution: modifying the description of one field can impact the performance of other fields. Therefore, in DOX Premium, any adjustments to field descriptions should be followed by an analysis of all fields, not just those that were changed. This iterative process continues until we achieve satisfactory results.

Steps for Iterative Prompt Development:

  1. Identify the field: Decide which field or specific extraction requires adjustment.
  2. Write or refine the prompt: Create a clear, specific base description.
  3. Review the results: Analyze the output for accuracy.
  4. Perform error analysis: Identify mistakes and discrepancies.
  5. Repeat: Iterate until the desired result is achieved.

 

Best Practices 

Leverage Default Extractors

Before diving into custom prompt development, take advantage of the default extractors provided by DOX Premium. These extractors are designed to select the best possible model for any given scenario—whether it's one of our own pre-trained models or a GenAI model. Trained on hundreds of thousands of business documents, they are highly accurate for common fields and document types, making them a valuable resource for initial extraction efforts. It's beneficial to check the performance of these pre-trained models, especially when the language is supported by them.

Activating these default extractors before relying on generative AI can help you assess whether further customization is needed. In many cases, combining pre-trained models with GenAI-based extractions yields optimal results for specific use cases. Here's a quick guide on how to activate the default extractors:

ChristophBatke_0-1719992471099.png

Clear and concise field descriptions

Using clear and concise language to describe fields significantly improves extraction accuracy in DOX Premium. Effective field descriptions help the model understand the specific data to extract, reducing ambiguity and errors. This is particularly important when similar fields exist, and you want to guide the model in a specific direction.

Field descriptions can also be tailored to manipulate the output format or language. In the following example from a delivery note, we compare a basic description with a more guided approach that provides clearer instructions. 

 

Don'tDo
Extract the countryExtract the country from the address. Please translate the country into english.
Extract the delivery conditionsExtract the delivery conditions. It is sufficient to extract only the abbreviation of the delivery condition mentioned on the document
 Material referenceMaterial Description

Extraction beforeExtraction beforeExtraction afterExtraction after

In the example, we clarified the country extraction and achieved the desired outcome of translating the country name into English. For delivery conditions, specifying that only the abbreviation should be extracted helped produce a more accurate result. The last example highlights that even semantically close descriptions can generate different outputs.
Not all business documents follow such a clear, labeled structure. For instance, many purchase orders include material numbers or codes that may not have explicit labels or indicators on the document. In these cases, understanding the entities—such as recognizing a specific format or consistent positioning on the document—becomes essential. Once this knowledge is gathered, you can improve extraction by incorporating these specifics into the field descriptions.

Schema description

DOX Premium now allows you to define schema descriptions, which serve as general instructions to our model before the schema and field-specific descriptions are applied. These schema descriptions can enhance the extraction process at the document level without focusing on individual fields.

One effective approach is to role-play by assigning a persona to the model. For example:
“You are an expert in extracting purchase orders from company X, which are related to products Y.”

This instruction can be followed by more general directives, such as specifying language preferences or emphasizing certain priorities. Schema descriptions can also help clarify correlations between fields. For instance, you can instruct the model to differentiate between the sending and receiving entities:
“Company X is always the receiver/supplier of the purchase order.”

In the screenshot example, we demonstrate a schema description being used to guide the model’s understanding of a nested structure within a delivery note. In this case, the line item table contains multiple material codes that may be linked to each other. The schema description informs the model to treat all nested material codes as part of a single line item:

“The material codes in the line item table can have a nested structure, meaning that subsequent codes can belong to each other. Please extract all the nested material codes as one line item.”

Extraction of nested structure with schema descriptionExtraction of nested structure with schema description

This approach helps the model accurately interpret and extract data even from complex or hierarchical document structures.

Regular Expressions

If you want to extract numbers that have a standardised format, it can help to use regular expressions for that in the field description. Regular expressions are like a smart search tool that helps you find specific patterns or sequences of text in a large amount of data. For example, if you are pretty sure that the article number on purchase orders has a structure of 'xxx.xx.xx', where x is a digit between 0 and 0, you can use the following regular expressions: \d{3}\.\d{2}\.\d{2}. As most of the DOX users (like myself) are no experts in this language, you can just use ChatGPT or other related tools to generate these expressions.

Field Names

It’s important to emphasize that field names are also part of the model input, which means they need to be meaningful and clear. As demonstrated in the schema description example, the wording of field names should align with the naming conventions used in the documents being processed. If users need to utilize different field identifiers for further processing, they can do so by using the 'Label' cell in the schema.

For more practical examples of field names and other related topics, you can check out the DOX help page: https://help.sap.com/docs/document-information-extraction/document-information-extraction/extraction...

Summary

With DOX Premium, end users have the flexibility to get creative and guide the model in a desired direction. Throughout the process of integrating this solution, it is crucial to align the model's behavior with the specific needs of the business. Since extracted data is often processed further in the workflow, ensuring its accuracy is essential.
Regular feedback loops between business and IT teams are invaluable for improving the service's usability and overall value. These ongoing interactions help to continuously refine field and schema descriptions, ensuring the solution delivers consistent and reliable results.

 

New Features

Instant Learning

The instant learning feature leverages a few-shot prompting approach to significantly enhance the AI model's understanding through real-time user feedback. In a few-shot prompting approach, the AI model is provided with several examples to help it better understand and execute a particular task. This method proves particularly effective for improving accuracy in complex tasks. For example, in sentiment classification, the model's output can be refined by supplying examples of the desired result:

 

 

 

 

 

This is awesome! // Negative
This is bad! // Positive
Wow that movie was rad! // Positive
What a horrible show! //

 

 

 

 

 

Users for now need to select the Setup Type 'auto' and exclude the default extractors to activate the feedback mechanism. Once enabled, they can review and correct the extracted values. This feedback, which functions as a few-shot prompt, helps the system learn about the document layout and the corrections. The knowledge gained is then used to improve extraction accuracy for future tasks with similar document layouts.

In the screenshots below, you can see the process in the DOX UI, where I selected a simple example of a naive address definition and made adjustments to refine the extraction for the address field:

Edit the documentEdit the document

Apply and confirm the correctionsApply and confirm the corrections

Next incoming (similar) documentNext incoming (similar) document

The last screenshot shows the updated extraction when uploading the document again with the same schema. The model now extracts the correct address from the top of the document rather than the shipping address. This change persists as long as the confirmed document remains stored in the backend, ensuring consistent extractions for similar future documents.

Data Type: List of Values

The new "List of Values" data type enables classification scenarios within DOX Premium and can significantly improve extraction performance for specific fields. This feature allows users to define expected output fields for extraction, which helps guide the model in its predictions.
For instance, if a user needs to classify documents as either invoices or debit notes, they can define those values in the list. The model will then analyze the entire document to make an informed classification. Below is an example of how this field is set up in the DOX UI:

Screenshot 2024-09-09 at 11.49.55.png

In this case, you can also leverage the field description and the description of the list values to provide additional instructions or hints. Another possible application of this feature is field-level classification, such as identifying the unit of measure. Here, users can define a list of possible outputs, like Kilograms or Tons, which the model will reference during extraction. 

 

Conclusion & Next Steps 

We hope this blog has demonstrated some useful hacks that you can apply to your next DOX project or use case. We would love to hear your feedback on how these techniques are working with your specific documents, or if you have additional ideas on how to improve the extraction process.
If you're still unsure whether DOX Premium is the right solution for your document needs, please don't hesitate to reach out to me or my colleagues on the AI RIG Team. We offer dedicated support to help activate the service and enable end users to successfully implement it. Additionally, we can review your documents and conduct a small Proof of Concept (PoC) using 10-20 documents, along with a live demo of the results.

christoph.batke@sap.com

sap_ai_rig@sap.com

2 Comments