Advanced Test Orchestration and Tester assignment ...

reiner_markheiser · ‎2023 Aug 09

In this blog post I would like to provide some details about using test plans for advanced test orchestration in SAP Cloud ALM. The blog will address the following questions:

When do you need test plans in SAP Cloud ALM?

How to use test plans and tester assignment in SAP Cloud ALM?

What are the differences compared with the Test Plans in SAP Solution Manager?

1. When do you need test plans in SAP Cloud ALM ?

In larger implementation or transformation projects you might want to run your test activities in iterations and so need setup multiple test cycles re-using the same or a similar set of test cases.

On top, you might want to roll-out the implemented solution or a global template to multiple organizations like subsidiaries, plants, markets or company codes which require dedicated test runs. Thus, you might want to re-use most of the test cases, manage and execute them within their own context.

You would like to specify in advance, who should execute which test case per Test Plan.

Although it was possible without test plans to run the same test case again while creating new test runs, the overall test status was always set to the status of the last test run.

Note:

As soon you set a test case to 'Prepared' you'll get one occurence of this test case in Test Execution. Adding this test case to a test plan and set this to 'In Testing' will create another occurence. So if do not want to have duplicate occurences in Test Execution, you do not need to create a test plan and can run your 1st test cycle without using test plans.

With the following illustrations I would like to explain the underlying concept with the existing approach without test plans and the new approach while using test plans:

1.1. Lean Test Orchestration Without Test Plans

Each test case appears in the Test Execution app as soon the status is set to ‘Prepared’

When the execution is started, a test run is created, status can be set, and defects created

In the following example, the first test run was set to ‘Passed’ after all test actions were completed successfully and the test run was set to ‘Finished’. Then a second test run was started and test action 1.1 was set to ‘Fail’. In consequence, the overall test status is displayed as ‘Failed’.

Test case with two test runs.

After introducing test plans, this lean approach is still available, so you could continue without any overhead. In most cases, this is sufficient for smaller projects and public cloud implementations with an agile or lean implementation approach. Therefore, we still support executing test cases without the need for test plans.

All prepared test cases have been completed.

1.2. Advanced Test Orchestration with Test Plans

For more complex projects, customers asked for the possibility to have a dedicated execution context for each test cycle.

To achieve this, you could create a test plan and assign relevant test cases

A subset of test cases have been added to the test plan that is still 'In Preparation'

In addition, you can assign a tester to a test case to define who should execute it.

Testers have been assigned to the test cases

As the test plan status is still 'In Preparation', no occurrences of the assigned test cases are displayed in the Test Execution app yet.

After switching the test plan status to 'In Testing', all assigned test cases in status 'Prepared' get a new occurrence (execution context) and the test execution can be started. Tester can now filter on assigned test cases.

The test plan is now 'In Testing', so assigned and prepared test cases can be executed in their own execution context

When you create another Test Plan with the same or similar set of Test Cases, you could have different tester assignments.

Another test plan has been created and is 'In Preparation' with a similar set of test cases but different testers assigned

The next picture shows the situation after Test Case 3 has been set to 'Prepared' and Test Plan B has been set to 'In Testing'.

Setting the new test plan to 'In Testing' will show related test cases with a new occurrence in status 'Initial' to start a new test cycle. Test Case 3 is now available for execution in three contexts (Test Plan A, Test Plan B, none)

2. How to use test plans and tester assignment in SAP Cloud ALM?

2.1. New Test Plan App

There is a new tile in the launch pad called Test Plans:

New tile 'Test Plans' in the SAP Cloud ALM launch-pad

When you open the Test Plan app, you will see a list of existing test plans with their status, time frame, and the aggregated preparation status and execution status of assigned test cases:

Test Plan Overview

You can create new test plans choosing 'Create' to get this detail screen in draft mode:

Detail screen for new test plan

To assign test cases choose 'Assign Test Cases' to get this simple assignment dialog:

Value help for test case

You can either start typing in the field to get a prefiltered list of test cases, or you could open the value help to get an advanced assignment dialog with search and filter options:

Advanced assignment dialog with search and filter capabilities

After confirming the selection and the assignment dialog, you will come back to the test plan details in draft mode, showing the assigned test cases:

Test plan details in draft mode with assigned test cases

In draft mode, the test plan is stored but not yet available to work with. Only when you choose 'Create', there will be a 'saved version' that can be used for processing.

Whenever you open the test plan in edit mode, a draft version is created, which contains all your changes without saving. You can switch between the draft and saved version of your test plans next to the test plan title:

Draft handling for test plans

You could even close your browser without losing your changes. You can find more details on draft handling here.

Assignment of a tester to a test case is available since December 2023 and can be used to define per test plan, which user should execute a certain test case.

Assign tester to test case

Besides the test plan statuses 'In Preparation' and 'In Testing', there is a status called 'Finished', which indicates that the test phase / test cycle is finished. This status does not have any impact on the visibility of related test cases occurrences in the Test Execution app.

2.2. Enhancements in the Test Execution App

In the Test Execution app, you will find a new filter for test plans and the occurrences of test cases are grouped by test plans.

Without setting the filter for test plans, you will see at least one entry per prepared test case belonging to the group '(None)' . In addition, you'll get another entry for each test case per test plan that is in status 'In Testing' or 'Finished'.

Since December 2023 a new filter is also available for tester to easily find the Test Cases assigned to the specified user (typically the current user).

Test Execution list view with filter set for assigned tester

You could also filter for multiple users. This allows to display test cases assigned to your colleagues or team and to execute tests on behalf of your collegues.

When planning to work with test plans, you could simplify the entry for your testers by setting the filter for the right test plan(s) and then sharing the URL.

A new Test Plan Status filter has been introduced in Q4 2024 to simplify access to relevant test cases. Further details see blog post:

Introducing the new "Test Plan Status" filter in Test Executions app in SAP Cloud ALM

2.3. Enhancements in the Test Preparation App

In the Test Preparation app, we have also added the filter for test plans to be able to reduce the list of test cases of the whole project to test cases that are not assigned yet to any test plan, or to list all test cases already assigned to a specific test plan.

Test plan filter in the Test Preparation app

2.4. Enhancements in Analytics for Test Plans

Since December 2023 you can find filters for test plans in relevant Analytics apps

Add Test Plan to the filter bar in Overview Page

Add Test Plan filter to your Overview Page

As a result related status information is filtered based on the assignment to Test Plans

Updated status information based on the selected Test Plan(s)

Add Test Plan to the filter bar in Traceability apps

Add Test Plan filter to your Traceability reports

As a result the data in the traceability apps will be filtered by assignment of Test Cases to Test Plans and related Requirements:

Example: Requirement Traceability with filter on certain Test Plan

These filter settings can be stored in a user defined view:

Save settings e.g. filter in a user specific view

3. What are the differences compared with the Test Plans in SAP Solution Manager?

In SAP Solution Manager it was mandatory to create a Test Plan, a Test Package and to assign a Tester to be able to execute a test. Many customers, working in a more agile approach treated this as an overhead. So, the goal with SAP Cloud ALM was to get rid of unnecessary complexity but providing more flexibility and integrated implementation and test approach. To make use of this, you should not try to copy your test concept from SAP Solution Manager, but to rethink your concept following these aspects:

Use SAP Cloud ALM entities like Scopes, Solution Processes and Tags to organize your test cases.

Use Requirements / User Stories and link related test cases to manage acceptance and functional tests.

Use Test Plans to orchestrate multiple test cycles for regression tests or for multiple rollouts of the same processes with the possibility to assign Tester directly to Test Cases within a Test Plan.

Rethink the size of Test Plans. Rather having higher number but smaller size of Test Plans compared to SAP Solution Manager so no additional level like Test Packages is required.

Use all dimensions and entities mentioned above for orchestration, execution and out-of-the-box reporting in Overview Page and Analytics apps.

Outlook on Planned Enhancements*

We plan* to enhance the test plan capabilities in further iterations:

Analytics:

Filter on test plans in Defects Reporting

Test Case Assignment to Test Plans

Advanced test case selection with additional filters in the Test Plan app

Tester Assignment:

Assign testers to test cases without test plan

Copy of Test Plans

Copy Test Plan to reuse assigned test cases and testers for another test cycle

Test Sequences:

Define the sequences of test cases within a test plan

Test Plans to Release assignment

Link several Test Plans to a Release to get an aggregated reporting on Release level.

Roadmap for Further Planned Innovations:

You can find the latest updates on our planning in the Roadmap Explorer for SAP Cloud ALM

* The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related document, or to develop or release any functionality mentioned therein. This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross negligence. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.