Legal Disclaimer
The following article deals with a mixture of legal as well as business requirements. Thus, it is in general infeasible to provide legally compliant and universally correct statements for all possible use cases. This article does not provide any legal guidance, nor should it be interpreted as giving it as such.
Executive Summary
This article serves to increase the understanding and should be a work of reference for the topics of end-of-purpose determination and ILM-based archiving. The terms usage, blocking, deletion, waiting and retention period are explained and put into context. In the course of doing so, various constellations of these time periods are analyzed. The analysis leads to practical consequences for usage.
Acknowledgments
I would like to give special thanks to my colleagues Reinhard Scheid, Manuel Brand and Paul Townsend. Their comments, reviews and knowledge corrected, improved and smothend this article.
Preface
I should point out that the following introductory words are to be taken with a good dose of humor. The group of topics this article deals with is primarily complex. Newbies should not consume this article all at once. Allow yourself breaks. Read each single chapter with nice gaps in between. And don't despair if you quickly feel like you are 'lost in space' - stick with it, it'll make sense in the end.
Chapter 1 of the General Data Privacy Regulation (GDPR) defines priciples for processing of personal data.
Personal data shall be [...] collected for specifed, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes. (Art. 5 p. 1 GDPR)
These principles result in the necessity to erase personal data when the so-called purpose has ended. An example:
Customer Miller puts in an order. Three days later, this order has not yet been processed. Miller revokes his order. During these three days, the order plus all necessary data are allowed to be stored. The primary purpose 'Order-Sales-Delivery' is active during these three days. After those three days, this primary purpose has ended. Thus, the personal data (the order plus any related personal data gathered in addition) needs to be deleted.
Deleting (personal) data directly at the point when the primary purpose expires is usually prohibited because of other laws. The most prominent example for the jurisdiction of Germany is the Tax Code (German term: Abgabenordnung (AO)). The Tax Code defines retention obligations and thus prevents data deletion for a certain period of time. In this case Art. 17 p. 3b GDPR comes into play. If other laws prevent deletion, the (personal) data needs to be blocked until its final deletion. I.e. access to blocked data must be prevented for typical users. Only users with special authorizations (and due to legally compliant reasons) are allowed to display the data (e.g. a tax auditor). There are no concrete requirements mentioned in the GDPR regarding the ways or rather the technical approach needed for blocking. However, the measures that are used need to be effective and applied. Setting a blocking indicator (e.g. a boolean value) might be suitable. In combination with authorization checks, the blocking indicator forces an access and thus also a processing restriction.
As a result, blocking takes place after primary purpose expiration. The blocking lasts until the data is deleted. The retention period (according to the German Tax Code (AO)) lasts for the same duration.
Metaphorically speaking, the end of purpose is the arrowhead of the yellow arrow.
Strictly speaking, it is a determination of a (calendar) date. In our example of the cancelled order it is the cancellation date. Depending on the business transaction, the so-called end of business can differ. E.g. for a non-cancelled order there will be no cancellation date, rather the delivery date might be used instead. Please be aware that the end-of-business determination within SAP software is not only product specific but in addition module specific, document-category specific etc. Thus, it is highly specialized.
The end-of-business determination for transactional data (e.g. orders, billing documents, …) is business-process specific. For master data, this determination is far more complex. Terminology already indicates that a direct transfer of the concept for transactional data is impossible. For master data it is more suitable to use the term end of purpose.
For master data, it is essential that there is no ‘best-before date’ defined by a process or by something else. Let’s take the example of a customer (KNA1
-record). Such a customer exists. Until when? Forever, without data privacy or other legislation. There is no process or organizational measure which limits the usage period and/or forces deletion of customer master data. Due to the GDPR, such behavior is made impossible. A customer master record must only exist as long as there are no expired primary purposes. The end of purpose for a (person-related) master record (blue dotted line in the figure shown) in general results only from the end dates of all business transactions (yellow arrows) related to this master record.
From a mathematics perspective, the end of purpose of (person related) master data records can be described as follows:
EoBPartner = max( EoBBus.Process1, EoBBus.Process2, ..., EoBBus.ProcessN)
In human words: The end date of the primary purpose for a business partner results from the set of business transactions of this partner, namely the maximum end date of all these transactions. Partner in this case is synonymous for SAP Business Partner (BUT000
), customers (KNA1
), vendors (LFA1
) and contact persons (KNVK
).
For the determination of a partner’s end of purpose, the following process applies:
This determination process has a major disadvantage: There is a fixed and direct coupling between the business transaction’s end dates and the partner’s end of purpose. In particular, there is no possibility to incorporate some kind of waiting period for this (over-)simplified approach. Hence, it is not taken and implemented in SAP products.
However, this (over-)simplified approach does show the basic principle of end-of-purpose determination without digression and (too) complex details. It should be understood before moving on to the more complex details that now follow.
OK, time for a coffee or tea break
Well-informed readers might have already related the term waiting period to the so-called residence period from ILM. For readers who are new to the topic, as well as those who have already fallen out with the ILM nomenclature - and all other readers for that matter - it is advisable to stick with the term waiting period for educational reasons. Strictly speaking, the ILM residence period is only one occurrence of a waiting period for the end-of-purpose determination.
The Pareto principle (80-20-rule) applies to the end-of-purpose determination too. The (over-) simplified approach explained so far only requires 20% of the cognitive and implementation effort to reach 80% of the overall solution. Let’s now tackle the remaining 20%, which unfortunately requires 80% (or more) of the cognitive performance and effort.
From the technical perspective, the non-existence of a waiting period is not an issue for the determination process. However, from operational perspective, such a waiting period might be required for the following reasons:
These different motivations for needing a waiting period typically result in different waiting period durations, e.g. depending on the process in question. In addition, it needs to be possible to define a waiting period using different criteria. Several overlapping waiting periods might be necessary too. To visualize this, let’s go back to the example we used previously:
Customer Miller puts in an order. This time it is not cancelled after three days but processed and delivered as usual. With the delivery, the invoice is issued. Customer Miller pays a few days after receiving the goods.
Assuming the invoice is the final business transaction for customer Miller in this process (i.e. for reasons of simplicity we won’t consider accounting or similar processes here), the (over-)simplified determination approach produces an end of purpose which is equal to the invoice’s end date. However, waiting periods should be applied too for the reasons already mentioned above. We therefore define the following waiting periods:
Type of Business Process | Waiting Period |
Order | 1 week |
Delivery | 8 weeks |
Invoice | 4 weeks |
The schema of business transactions changes as follows: There is still a direct succession of business transactions. However, the end-of-purpose determination now considers these waiting periods.
It should be noted that depending on the waiting periods (which might be nonsensical), some other business transaction could also be controlling the end of purpose.
OK, time for a relaxation exercise
So far, we only distinguished the waiting period based on the type of business transaction involved. In general, there is nothing stopping you from further differentiating within a type (only the enormous increase in complexity). For example, the delivery’s waiting period could be derived from the delivery type. Depending on the type of business transaction involved, there are specific criteria that are only available and/or relevant for this type. There might be some criteria, however, that apply to several types of business transactions. Let’s have a look at some concrete examples:
Criteria for defining waiting periods for orders:
Criteria for defining waiting periods for deliveries:
Criteria for defining waiting periods for invoices:
In this example, all three types of business transactions use the criterion sales organization. Similarities can be seen for the sales document category. One criterion, the sales district, only occurs for deliveries. We enhance the table for defining waiting periods as follows:
Type of Business Process | Order type | Sales Document Category | Sales Organization | Delivery Type | Sales District | ... | Waiting Period |
Order | * | * | DE | - | - | ... | 1 week |
Order | * | * | FR | - | - | ... | 2 weeks |
Delivery | - | * | DE | * | * | ... | 8 weeks |
Delivery | - | * | FR | * | * | ... | 9 weeks |
Invoice | - | - | DE | - | - | ... | 4 weeks |
Invoice | - | - | FR | - | - | ... | 5 weeks |
Explanation: Criterion not available ⇔ - | Criterion irrelevant ⇔ *
Due to the additional criteria, it is possible to define the waiting period specific to a sales organization (this is shown in the above table). The three original waiting periods are distinguished per country (DE, FR). In addition, the waiting period for country FR is always one week longer compared to the one for country DE.
Table 2.2 also shows a clear disadvantage. Nearly all columns (criteria) are only used by a single type of business transaction. Only few are used multiple times. Thus, this is a sparse table (refer to sparse matrix). Adding more types of business transactions with additional criteria would vastly increase the size of the table. At the same time the table’s usability would tend towards zero. Apart from the poor usability, there are also technical disadvantages for such a table concept:
A solution for such sparsity is to split the single table into several ones. Due to the different types of business transactions available, it is advisable to use one table per type. This enables you to add further criteria per type in future or to implement extensibility features per type.
Order Type | Sales Document Category | Sales Organization | Waiting Period |
* | * | DE | 1 week |
* | * | FR | 2 weeks |
Explanation: Criterion irrelevant ⇔ *
Delivery Type | Sales District | Sales Document Category | Sales Organization | Waiting Period |
* | * | * | DE | 8 weeks |
* | * | * | FR | 9 weeks |
Explanation: Criterion irrelevant ⇔ *
Billing Type | Sales Organization | Waiting Period |
* | DE | 4 weeks |
* | FR | 5 weeks |
Explanation: Criterion irrelevant ⇔ *
The used schema changes as follows:
The waiting period is offset from the blocking period. This might be surprising, but the explanation is as follows: Firstly, the waiting period cannot be part of the usage period as the latter ends with the business transaction’s intrinsic date. Secondly, according to §147 AO DE/EN there is no relation between the retention period and a waiting period . The retention period (in Germany) is just determined using the creation date (cf. §147 p.4 AO DE, EN).
Some readers might have already noticed that the relationship between master and transactional data has not been differentiated clearly in the previous figures. The following figure illustrates the periods for usage, waiting, blocking and deletion separately for master and transactional data. Let’s start by looking at the order again. Its figure changes as follows:
We enhance this figure with customer master data that we haven’t visualized up until this point. The usage period of the business partner master data record (green arrow) is by definition equal to the order’s usage period plus its waiting period. At the end of the order’s waiting period, the order itself as well as the business partner master data record are to be blocked.
For auditing and tax auditing, it is not sufficient to only retain the order. The business partner needs to be retained too. The retention period of the business partner master data record is determined by the order’s end of purpose as well as the order’s retention period. Both periods end at the same point in time (or rather on the same day), since the business partner master data record must not be deleted before the order is deleted. In the case here (we’re only considering the order), the blocking (period) of the business partner master data record comprises the same period as the blocking of the order. Moving on from that, now we want to add the delivery and invoice to the figure to get the big picture. First, we add the delivery to the figure.
As the difference to the previous figure shows, considering the delivery in addition changes the business partner master data record’s end of purpose. It has now moved into the future. In addition, the retention period for the business partner master data record has changed. It starts with the end of purpose that results from the order and the delivery, and ends with the maximum retention period of the order and the delivery. Let’s add the invoice.
Again, the end of purpose for the business partner master data record has changed. Now it results from the dates of all business transactions (order, delivery, invoice). The business partner master data record’s retention period aligns with the maximum end date of all three retention periods for order, delivery and invoice.
The examinations of the waiting period made so far are too coarse-grained. Apart from the end-of-purpose determination we have considered up to now, a waiting period is also present for data archiving. Overall there are the following waiting periods:
The following table includes information about each definition location, definition details, and the commonly used synonym for all three waiting periods.
Waiting Period | Location of Definition | Definition Details | Common Synonym | Usage |
K1 | ILM | • Transaction IRMPOL • Policy category residence periods (RST) • Audia area (always) BUPA_DP • Arbitrary policy name • ILM objects for SAP Business Partner (CA_BUPA), Customer (FI_ACCRECV), Vendor (FI_ACCPAYB), Contact Person (FI_ACCKNVK) | residence period | Residence policies in audit area BUPA_DP: determine the waiting period for the EoP-check |
K2.1 | Application customizing table | • Via Customizing reference guide or SM30 • Application-specific table | residence period | determine the waiting period for archiving |
K2.2 | ILM | • Transaction IRMPOL • Policy category residence periods (RST) • Audit area (always) ARCHIVING • Arbitrary policy name • ILM objects for Sales Order (SD_VBAK), Delivery (RV_LIKP), Invoice (SD_VBRK) | residence period | Residence policies in other audit areas: determine the waiting period for archiving |
Looking at the table above, it’s obvious that the usage of the term ‘residence period’ is often a cause for misunderstandings, since the same term is used for all three waiting periods. However, all three waiting periods control different aspects. Thus, they should be looked at independently from each other. In addition, either waiting period K2.1 or K2.2 should be used, but not both at the same time. For this reason, only ILM-based alternative waiting period K2.2 is considered in the following section to begin with.
Technically, there is nothing stopping anyone from using all three waiting periods at the same time. It would be possible to define a waiting period of 1 week for the purchase order for the end-of-purpose determination of the business partner master data record (K1), and at the same time a waiting period of 1.5 weeks for data archiving purchase orders (K2.2). The figure for this would look like this:
The issue with this configuration is the existence of the pink-colored time period. Within this space of time, the following is in force:
Now let’s examine the opposite case. In this case, the order’s waiting period for the end of purpose of the business partner master data record (K1) is defined as 1.5 weeks. At the same time, the waiting period for data archiving of orders (K2.2) is only 1 week. The figure looks like this:
The issue with this configuration is once again the existence of the pink-colored space of time. Within this period the following applies:
The figure for K1 = K2.2 is as follows:
SAP Sales & Distribution (SD) Sales Order
So far, we have not yet examined waiting period K2.1. The reason for this is a property of this waiting period. By definition, it is application-specific and it is (potentially) interpreted non-uniformly. For example, the SD sales order and the SAP Customer Relationship Management (CRM) sales order have been using waiting period K2.1 for decades. However, the related data archiving processes differ when it comes to the technical check for whether or not the business (respective) order is archivable.
For the SD sales order the technical and operational checks on the termination of waiting period K2.1 are performed within a single process step (refers to the archive write program). This step also checks waiting period K2.2 (if defined). If the evaluation is successful, data archiving is directly possible.
The following configurations are possible:
K2.1 = K2.2
No divergent behavior compared to K1=K2.2 (see chapter 3.2).
K2.1 < K2.2
As both K2.1 and K2.2 are considered, only K2.2 is significant. Thus, there is no divergent behavior compared to K1 = K2.2.
K2.1 > K2.2
As both K2.1 and K2.2 are considered, only K2.1 is significant. As a result there is no divergent behavior compared to K1 = K2.2
After examining all possible configurations, it can be summarized that for the SD case either K2.1 or K2.2 should be used, but not both waiting periods at the same time. Moreover, these two waiting periods have the same effect and are thus equivalent.
CRM Sales Order
For archiving CRM-business transactions there are different possibilities. For the CRM sales order alternative A performs technical and operational checks as well as checks for the expiration of waiting period K2.1 within a single process step. However, this process step is a preliminary step (the so called archive preprocessing program). The successful check results in a status change of the sales order (can be archived (refer to transaction BS23 and status I1100)). The business transaction is not automatically archived. Another process step is necessary which completes archiving. This second process step checks waiting period K2.2.
In contrast to the SD sales order the waiting periods K2.1 and K2.2 are not equivalent but different and cumulative. In addition, the CRM sales order is not changeable during waiting period K2.1 and K2.1 is one day at minimum (cannot be zero). The SD sales order is still changeable till the begin of the retention period. Hence, the waiting periods K2.1 and K2.2 are mutually exclusive for many CRM-business transactions (refers to One-Order-based business transactions. Non-One-Order based business transactions often follow different rules during archiving.). From operational perspective both waiting periods address the same requirements. Thus, one could conclude to define K2.1 and K2.2 to have an equal length. Due to their implementation and their resulting flow of both waiting periods *after each other one should - if possible - relinquish using waiting period K2.1 (K2.1= 1 day).
Alternative B avoids the application specific waiting period K2.1 completely. For this purpose a different, alternative process step is used. In stead of setting status archivable the status archiving direct deletion (refer to transaction BS23 and status I1108) is used.
The following configurations are possible:
K2.1 = K2.2
The end of business transaction for the sales order is already reached (corresponds to the existence of status I1005 completed at the business transaction). The sales order's retention period has not started yet because K2.1 has not expired. This results in the problematic aspects described in chapter 3.2.
K2.1 < K2.2
This configuration is further distinguished into:
K2.1 = 1 day
With this configuration alternative A as well as B are defined suitable. No divergent behavior in comparison to K1=K2.2 occurs (see chapter 3.2) except for the one day.
K2.1 > 1 day
For alternative *B there is no divergent behavior in comparison to K1=K2.2 (see chapter 3.2). However, for alternative A there are the problematic aspects described in chapter 3.2.
K2.1 > K2.2
There are the problematic aspects described in chapter 3.2.
Besides the logical implications for waiting periods that are described in the previous chapters there are legal boundary conditions for waiting periods. Depending on the relevant jurisdiction a company must adhere to one or more accounting regulations. For example the International Financial Reporting Standards (IFRS) as well as the regulations of Handelsgesetzbuch (HGB) might be authoritative. In such a case it might be likely that waiting periods K1 and K2.2 are four years. A waiting period of such a length is obviously longer than the timer period of the active usage of a business transaction. Anyways the discussed logical implcations still apply for such long waiting periods.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
22 | |
11 | |
10 | |
9 | |
8 | |
6 | |
6 | |
6 | |
5 | |
5 |