Creating a Web-Application for SAP Data Intelligence

[ThorstenHa]

Introduction

Sometimes you need just a small web-application interacting with SAP Data Intelligence without having to login into SAP Data Intelligence, launch an app and filter and select what you actually like to do. Due to the fact that SAP Data Intelligence has open APIs listed on the SAP API Business Hub  and an easy to use web-app deployment feature on SAP Business Application Platform this could be easily done in less than an hour. In the following I show you how to do this with following short example:

List all my pipelines to check if a pipeline has completed or is still running.

For this task we need to

1. know the RestAPI definition of how to get all the pipelines,


2. write a web-page that calls the RestAPI and presents the result


3. have a space and quota on Cloud Foundry of SAP Business Transformation Platform for deploying the application

 

RestAPIs of SAP Data Intelligence Modeler

On the SAP API Business Hub we get all the RestAPIs definitions. As an alternative you could get the swagger-definitions from "<DI-URL>/app/pipeline-modeler/swagger.yaml". There we finally find the RestAPI that lists all your pipelines listed in the runtime environment:

List RestAPIs

 

We have got the location: /v1/runtime/graphs that we need to add to compose the whole RestAPI-ULR:

-> <URL of your DI-system>/app/pipeline-modeler/service/v1/runtime/graphs 

(example: https://vsystem.ingress.dh-xxxxx.dhaas-live.shoot.live.k8s-hana.ondemand.com//app/pipeline-modeler/s...

For a GET we do not need to provide any parameter and the definition tells us that the response will provide us a JSON-string. The details you find in the swagger-documentation.

For testing you could use the POSTMAN-app, where in addition to the URL you could enter your credentials. If it runs into an error the reason might be a "CROSS-DOMAIN" error and with the additional header-item: 'X-Requested-With': 'XMLHttpRequest' you might solve it.

 

Web-App

For the web-application I am using Python with the Flask-package because as a passionate python programmer it comes quite natural.

app.py

# Python package for import



import json

from datetime import datetime



from flask import Flask, render_template, flash

from flask_bootstrap import Bootstrap

from flask_wtf import FlaskForm

from wtforms import StringField, SubmitField

from wtforms.validators import DataRequired

from wtforms.widgets import PasswordInput

import requests



# FLASK init with Bootstrap CSS

app = Flask(__name__)

bootstrap = Bootstrap(app)

app.config['SECRET_KEY'] = 'Di4PipeMon2021'



# WTForm 

class requestForm(FlaskForm):

    di_url = StringField('DI-URL: ', validators=[DataRequired()])

    tenant = StringField('Tenant: ', validators=[DataRequired()],default='default')

    user = StringField('User: ', validators=[DataRequired()])

    pwd = StringField('Password: ', widget=PasswordInput(hide_value=False), validators=[DataRequired()])

    verification = BooleanField('TLS Verification', default=True)

    submit = SubmitField('Submit')





# Binding of 'index'-page

@app.route('/', methods = ['GET'])

def index():



    requestform = requestForm()

    data = list()



    if requestform.validate_on_submit() :

        response, data = call_restapi(user = requestform.user.data,

                                      pwd = requestform.pwd.data,

                                      url=requestform.di_url.data,

                                      tenant= requestform.tenant.data,

                                      verification = requestform.verification.data)



        if not response == 200 :

            flash("Response: {}".format(response),'warning')

            return render_template('request.html', dictlist = data, form=requestform)



    return render_template('request.html',dictlist = data, form=requestform)





# RestAPI call

def call_restapi(user, pwd, url, tenant) :

    # send request

    auth = (tenant+'\\'+user, pwd)

    #headers = {'X-Requested-With': 'XMLHttpRequest'}

    headers = {}



    resturl = url + '/app/pipeline-modeler/service/v1/runtime/graphs'



    resp = requests.get(resturl, auth=auth, headers=headers,verify = verification)

    data = list()



    if resp.status_code == 200 :

        rdata = json.loads(resp.text)

        for r in rdata :

            started = datetime.fromtimestamp(r['started']).strftime('%Y-%m-%d %H:%M:%S') if r['started'] >0  else '-'

            submitted = datetime.fromtimestamp(r['submitted']).strftime('%Y-%m-%d %H:%M:%S') if r['submitted'] > 0 else '-'

            stopped = datetime.fromtimestamp(r['stopped']).strftime('%Y-%m-%d %H:%M:%S') if r['stopped'] > 0 else '-'

            data.append({'Pipeline': r['src'], 'Status': r['status'],'Submitted': submitted,'Started': started,

                         'Stopped': stopped,'Messages':r['message']})



    return resp.status_code, data



# APP CALL

if __name__ == '__main__':

    app.run('0.0.0.0',port=8080)

 

request.html

For generating the HTML-page I am using the template framework jinja that it the obvious choice when using Flask. In particular when you like to create a simple but dynamic table jinja can be of great help.

{% extends "bootstrap/base.html" %}

{% import "bootstrap/wtf.html" as wtf %}



{% block title %}Runtime Pipelines{% endblock %}



{% block head %}

{{ super() }}

<link rel="shortcut icon" href="{{ url_for('static', filename='favicon.ico') }}" type="image/x-icon">

<link rel="icon" href="{{ url_for('static', filename='favicon.ico') }}" type="image/x-icon">

{% endblock %}





{% block navbar %}

<nav class="navbar navbar-expand-lg navbar-light">

    <div class="container">

      <div class="navbar-brand"><H1><img src='/static/SAP_R_grad.png' style="height:50px" class="img-fluid" alt="SAP logo">Data Intelligence</H1></div>

      <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation">

        <span class="navbar-toggler-icon"></span>

      </button>

      <div class="collapse navbar-collapse" id="navbarNav">

      </div>

    </div>

</nav>

{% endblock %}



{% block content %}

<div class="container">

    {% block page_content %}

    <br>

    <br>

    {% with messages = get_flashed_messages(with_categories=true) %}

    {% if messages %}

        {% for category, message in messages %}

            <div class="alert alert-{{category}}">

            <button type="button" class="close" data-dismiss="alert">&times;</button>

            {{ message }}

            </div>

        {% endfor %}

    {% endif %}

    {% endwith %}



    <div class="page-header">



    </div>

    {{ wtf.quick_form(form) }}

    <br>

    <h2>Runtime Pipelines</h2>

    <table class="table">

        <thead>

        <tr>

                {% for hi in dictlist[0] %}

            <th scope="col">{{ hi }}</th>

            {% endfor %}

        </tr>

        </thead>

        <tbody>

        {%  for row in dictlist %}

        <tr>

            {%  for val in row.values() %}

            <td>{{ val }}</td>

            {% endfor %}

        </tr>

        {% endfor %}

        </tbody>

    </table>

    {% endblock %}

</div>

{% endblock %}



{% block scripts %}

{{ super() }}

{% endblock %}

 

Business Transformation Platform Deployment

For deploying the web-application you first need to create a couple of files that helps to deploy the final applications:

• Procfile (defines how to call the application)


• runtime.txt (sets the script-language and version)


• requirements.txt (lists all necessary packages)


• manifest.yaml (contains the details for the deployment)

 

Proc

web: python app.py

runtime.txt

python-3.8.8

Not all python releases are supported but when you deploy the application you get the information if your chosen release is not supported together with a list of alternatives.

requirements.txt

Compare with import-statements in app.py.

Flask

Flask-Bootstrap4

Flask-WTF

jinja2

WTForms

Werkzeug

requests

manifest.yaml

Here you setup basically the memory you need for the application. Later when the application runs you can check in BTP how much is consumed actually and might adjust the settings.

---

applications:

- name: pipelinemonitor

  host: pipelinemonitor

  memory: 256M

  disk_quota: 256M

  timeout: 60

  buildpack: https://github.com/cloudfoundry/buildpack-python.git

 

Deploying

My preferred way of deploying a web-application on BTP is using the command-line version of the Cloud Foundry. For this you need to install cf CLI v6 first.

Then change to the directory of your app where all you files above are stored and login:

cf login 

The API-endpoint is already selected and you get a prompt for your user (email) and password. Due to the 2-factor authentication you might have to add after your password followed by your additional authentication code created by a separate Authenticator-app. You also might have to select your "org" and "Space" if you have more than one space.

Then you can push your application

cf push pipelinemonitor

When using the command-line option you see the progress of the installation and if something goes wrong you have mostly all the information you need to solve the issues. In case it is not enough you can download the logs in addition with the command:

cf logs --recent

 

In the BTP Cockpit you finally see the running instance of the web-application and its url:

BTP Cockpit

 

Result

When you call the application URL that the BTP cockpit provides, you get an application that looks like this one:

 

web-app pipelinemonitor

Summary

I have outlined how to use the Data Intelligence APIs listed on the SAP API Business Hub for a small web-application that serves a very specific use case. In addition you see how you can use the tools of the Business Transformation Platform to deploy the application.