Technology Blogs by Members
Explore a vibrant mix of technical expertise, industry insights, and tech buzz in member blogs covering SAP products, technology, and events. Get in the mix!
cancel
Showing results for 
Search instead for 
Did you mean: 
patrobibeksap
Explorer
974
Introduction:

This is going to be my first blog on SAP Data Intelligence and quite excited about sharing with all my SAP colleagues/members working on this tool.

This document overviews about deployment life cycle management for SAP Data Intelligence using CI/CD methodology. Since SAP DI doesn’t provide any proprietary tools for deployments as of the current DI version 2303.15.28.

Business Pain Points:

  • There is no SAP proprietary solution for deployment of SAP DI pipelines and operators using CI/CD approach

  • Exporting and Importing files into Production systems are restricted in some of the client deployments

  • Manual method of deploying DI objects is susceptible to human mistakes


Business Value Proposition:

  • Implementation of CI/CD throughout Development, testing and Production phases of SAP DI development.

  • Automatic deployment of DI pipelines and operators without any manual intervention

  • Review of the DI objects when pushed for deployment to check its completeness and inform developers/object owners about it.

  • Authentication based approach where the DI objects are reviewed and authenticated at every level to discard any unintended changes to get moved between different systems.


Deployment Workflow Path:

The following picture shows the path how the objects are moved starting from development till getting it deployed into Production.


The following points to be noted here:

  • Develop, Test and Master branches are protected so that no one can able to push to it directly.

  • The changes can be moved from Feature branches to develop, test and Master separately which helps us independently deploy DI objects in case required.

  • Each workflows are event based triggered like based on pull or push requests

  • Each workflow can identify and deploy only new DI pipelines or changes pipeline rather than picking all from the repository. This is the strength of this pipeline which avoids re-deployment of unchanged DI objects into test and production systems.


Design of deployment workflow

The workflows are designed using Git Action commands. These action commands are branch specific and execution is based on event (Push or Pull requests). This workflow can identify the DI pipelines/operator files which are modified or newly created and trigger deployment only for those objects.

The sample code here can be referenced for designing your deployment workflows.
name: Deployment of DI objects to DEVELOPMENT tenant

on:
push:
branches: [<Github development branch name>]

jobs:
setup:
runs-on: ubuntu-latest
outputs:
matrix: ${{steps.list_file.outputs.value}}

steps:
- name: check out repo
uses: actions/checkout@v2

- name: Get Diff Action
uses: technote-space/get-diff-action@v6
with:
format: JSON

- name: Filter files for graphs and operators only
run: |
res=$(echo '${{ env.GIT_DIFF }}' | jq '.[] | select(. | contains("graph.json") or contains("operator.json"))' | jq -Rsc '. / "\n" - [""]')
echo "filtered_op=$res" >> $GITHUB_ENV
shell: bash

- name: Passing the final output to build
id: list_file
run: |
res=$(echo '${{ env.filtered_op}}' | jq '.[]')
if [ -z $res ]; then
echo '::set-output name=value::["NO DEPLOYMENT OBJECTS FOUND"]'
else
echo '::set-output name=value::${{ env.filtered_op }}'
fi

build:
needs: [setup]
runs-on: windows-latest
strategy:
max-parallel: 1
matrix:
value: ${{fromJson(needs.setup.outputs.matrix)}}

steps:
- name: Checking out your current git repository
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
uses: actions/checkout@v2

- name: Capturing the solution name to bundle
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: echo "obj_name=$(echo ${{ matrix.value }} | awk 'BEGIN{FS=OFS="/"} {print $--NF}')" >> $GITHUB_ENV
shell: bash

- name: Finding the Source directory
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
echo "source_dir=$(echo ${{matrix.value}} | awk 'BEGIN{FS=OFS="/"}NF--')" >> $GITHUB_ENV
shell: bash

- name: Defining the target directory for solution
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
echo "target_dir=$(echo ${{ env.obj_name }}/content/files/vflow/${{env.source_dir}})" >> $GITHUB_ENV
shell: bash

- name: Building the solution directory structure
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
mkdir -p ${{env.target_dir}}
cp -R ${{env.source_dir}}/* ${{env.target_dir}}/
cp -R ${{env.source_dir}}/manifest.json ${{ env.obj_name }}/
shell: bash

- name: Log-in to DI through vctl
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: vctl\vctl.exe login ${{secrets.<DEV INSTANCE URL GIT SECRET>}} --user-cert ${{secrets.<bundle.pem GIT SECRET>}} ${{secrets.<key.pem GIT SECRET>}}

- name: Bundle the solution into zip file in current working directory
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: vctl\vctl.exe solution bundle ${{env.obj_name}}

- name: Capture the solution version post bundle
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
echo "zip_name=$(ls | sed -n 's/\.zip$//p')" >> $GITHUB_ENV
shell: bash

- name: Get list of solutions from the strategy
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
vctl\vctl.exe strategy get ${{secrets.<default DI strategy name in GIT SECRET>} -o "json" | Out-File ./input.json

- name: Look up for this solution version if available in the strategy
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
jarray=$(cat ./input.json | jq '.layers')
obj_arr=$(echo $jarray | jq -c -r '.[]')
echo "str_exist_flag=$(echo "NO")" >> $GITHUB_ENV
flag=0
for obj in ${obj_arr[*]}
do
val=$(echo $obj)
if [ "$val" = "${{env.zip_name}}" ]; then
echo "str_exist_flag=$(echo "YES")" >> $GITHUB_ENV
flag=1
fi
done
if [ $flag -eq 1 ]; then
echo "This solution ${{env.zip_name}} version already exists in strategy."
else
echo "This solution ${{env.zip_name}} version doesn't exists in strategy."
fi
shell: bash

- name: Remove the solution version from strategy if already exists
if: ${{ env.str_exist_flag == 'YES' }}
run: vctl\vctl.exe strategy remove ${{secrets.<default DI strategy name in GIT SECRET>}} ${{env.zip_name}}

- name: Get list of all solutions from the tenant
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
vctl\vctl.exe solution list -o "json" | Out-File ./solution.json

- name: Look up for this solution version if available in the tenant
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: |
obj_arr=( $(jq -c -r '.[] | (.Manifest.name) + "-" + (.Manifest.version)' solution.json) )
echo "sol_exist_flag=$(echo "NO")" >> $GITHUB_ENV
flag=0
for obj in ${obj_arr[*]}
do
val=$(echo $obj)
if [ "$val" = "${{env.zip_name}}" ]; then
echo "sol_exist_flag=$(echo "YES")" >> $GITHUB_ENV
flag=1
fi
done
if [ $flag -eq 1 ]; then
echo "This solution ${{env.zip_name}} version already exists in solution list."
else
echo "This solution ${{env.zip_name}} version doesn't exists in solutions."
fi
shell: bash

- name: Delete the existing solution version from tenant if already exists
if: ${{ env.sol_exist_flag == 'YES' }}
run: vctl\vctl.exe solution delete $(echo ${{env.zip_name}} | cut -d "-" -f 1) $(echo ${{env.zip_name}} | cut -d "-" -f 2)

- name: Uploading updated solution version to the tenant
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: vctl\vctl.exe solution upload ${{env.zip_name}}.zip

- name: Adding updated solution version to the strategy
if: ${{ matrix.value != 'NO DEPLOYMENT OBJECTS FOUND' }}
run: vctl\vctl.exe strategy add ${{secrets.<default DI strategy name in GIT SECRET>}} ${{env.zip_name}}

Expected learnings from this blog:

  • This is a ready to use template which can be reused in DI project deployments with CI/CD approach.

  • With the use of this approach, manual handling of deployments like import/Export can be avoided.

  • We can track the history of changes for any object using git commit history.


I encourage all readers to provide your comment and happy to hear back any suggestions/ improvements in this approach. This will help me to improve this deployment workflow.
4 Comments
Labels in this area