Using the Job Scheduling Service in SAP BTP [5]: L...

CarlosRoggan · ‎2020 Apr 23

This tutorial is part of a little series about the SAP Job Scheduling service.

Quick links:
Intro Blog
Tutorial 1
Sample Code (Trial)
Sample Code (Prod)
Quick Guide

In previous tutorials, we’ve learned how to configure the Job Scheduling service and how to write apps which are protected with OAuth 2.0.

Basically, what we’ve done was to create a web app with a REST endpoint and ask the Job Scheduling service to call that endpoint.
That endpoint hopefully returns a success response and the Job Scheduling service is happy to mark the jobrun with green letters.

In other words:
The Job Scheduling service WAITS for the response.
It is like a patient cab driver.... but his patience is not endless:
To be concrete: patience ends after 15 seconds.
Yes:
The Job Scheduling service has a timeout of 15 seconds.

But what can we do, if we have operations that require more time?

In this blog, let’s discuss an important scenario: Long running jobs
They are required in scenarios like e.g. data replication running overnight, etc.
In such case, the Job Scheduling service does NOT wait for the response of the called endpoint.
Such a scenario has to be executed in an ASYNCHRONOUS way.
This means:
The Job Scheduling service triggers an app’s endpoint.
Once the app is done, it calls a sort of a callback to inform the Job Scheduling service about the finished job.

Let’s learn how you can do this.

Prerequisites

You should have had a look at the previous tutorials.
You should be familiar with Node.js. Otherwise, check here.
You need an account in SAP Business Technology Platform (SAP BTP).

As usual, our tutorial is based on the Trial account ('lite' for the Job Scheduling service), such that everybody can follow it.
In a productive account ('standard' for the Job Scheduling service), the code looks a bit different, but this is explicitly described in a section below.

Overview

1. See the timeout.
2. Implement the required app behavior.
2.1. Endpoint
2.2. Backend operation
2.3. Update status
2.4. Update status prod
3. Test it.

1. Normal Synch Job

Let‘s start with the negative scenario, then we repair it in a second step.
We create a little node application which represents a long-running operation.
To simulate it, we just wait for 18 seconds.

1.1. Create the Job Scheduling service instance.

See description here.

1.2. Create an XSUAA instance.

See description here.

1.3. Create an app.

const express = require('express');
const app = express();

app.get('/runjob', function(req, res){
   setTimeout(() => {
      res.send('Finished job after 18 seconds waiting');
   }, 18000);
});

app.listen(process.env.PORT, function(){})

1.4. Deploy the app.

You know how to deploy your app.

1.5. Try it in a browser.

Invoke the endpoint in a browser window, e.g.
https://longrun.cfapps.eu10.hana.ondemand.com/runjob
After 18 seconds, you get the response with the message as coded above.

1.6. Try the Job Scheduling service.

Create a job, check the logs. There you'll find the error message:

Error: ESOCKETTIMEDOUT - Request timeout after 15 seconds

OK, this was expected because in our app, we wait for 18 seconds, which is longer than the allowed 15 seconds.
Yes, our wait-18-seconds is our long-running operation.
Let’s see the next chapter to learn how to repair it.

Summary:
The Job Scheduling service has a built-in timeout of 15 seconds.

2. The Solution: Async Job

What needs to be done?

1. Our endpoint has to respond immediately with success status 202.
2. We have to trigger our long-running operation.
3. We have to manually set the result in the Job Scheduling service.

OK, let’s see it in detail.

2.1. The Endpoint

The Job Scheduling service doesn’t know much about us.
It just knows the endpoint which it is supposed to call.
So we have to inform the Job Scheduling service about our plans of running long:
for that purpose we use our endpoint.
Concrete: Our endpoint has to send a response with status code 202.
Like that, we tell the Job Scheduling service not to wait for us.

The Job Scheduling service interprets the status codes as follows:
* Error code: endpoint operation had an error -> job is completed and will be marked with an error
* 200 : endpoint operation finished successfully -> job is completed with success
* 202: endpoint has been triggered and has responded with 202 -> job is not completed, it is on hold until further notice

As per definition, status code 202 means “Accepted”.
See appendix for explanation of status code 202.

For our endpoint implementation this means:
As soon as our endpoint is invoked, we respond with 202:

app.get('/runjob', function(req, res){       

   res.status(202).send('Accepted async job, but long-running operation still running.')

As you’ve already noticed, I’m distinguishing:

The term “job” belongs to the Job Scheduling service, it is the trigger in the cloud.
The term “operation” indicates the actual work, which is done e.g. in a backend.

2.2. The Backend Operation

After responding to the Job Scheduling service, our endpoint will trigger the long-running backend operation.
In our little sample app, we just wait for 3 seconds:

const doLongRunningOperation = function (doFail) {
   return new Promise((resolve, reject)=>{
      const letItFail = doFail
      setTimeout(() => {     
         if(letItFail === 'true'){
            reject({message: 'Backend operation failed with error code 123'});
         }else{
            resolve({message: 'Successfully finished long-running backend operation'})   
         }
      }, 3000); // wait... wait...
   })
}

Note:
To make testing easier, our little sample app can be configured:
Depending on a URL parameter (which is set by us), the long-running backend operation will fail or will succeed.

2.3. The Status

Now comes the interesting part:
How to inform the Job Scheduling service about the result of our backend operation?
We cannot expect that the Job Scheduling service calls us again to ask us.
So now it is vice versa: WE have to call the Job Scheduling service.

How do we call the Job Scheduling service?
The Job Scheduling service provides a REST API for remote control.
This is a big feature and here we cover a small subset:
How to use the REST API to set the status of a job execution?
See the REST API documentation for an overview of other features (create jobs, view results, etc).

In order to access a particular job execution and set the status, we need some information:
1. We need the internal info about the job and the URL to call.
2. We need authentication info.
3. We need to know the structure of the endpoint and the data which we have to send.

And this is how we get the required info:

2.3.1. How to get the internal info of a concrete job to update?
This information is sent to us by the Job Scheduling service when it calls our endpoint.
It is hidden in the headers.
And here are the headers which we need:

Use case	Header name
Which job?	x-sap-job-id
Which schedule?	x-sap-job-schedule-id
Which particular job run?	x-sap-job-run-id
Which URL to call?	x-sap-scheduler-host

Example values:

x-sap-job-id	12345
x-sap-job-schedule-id	cfca1190-faf5-476d-bc03-b70bfa82dba3
x-sap-job-run-id	b2cde24e-5479-46da-b3a1-1f1da93823d0
x-sap-scheduler-host	https://jobscheduler-rest.cfapps...hana.ondemand.com

2.3.2. How to authenticate?
There are credentials for us, we only need to find them.
As usual in SAP BTP, our app gets credentials in the application environment when it is bound to a service instance.
We can see it either in the SAP BTP cockpit, or in the command line (as described in the previous blog).
Now we need to distinguish:
Authentication mechanism is different in Trial and productive landscape, or in other words, it is different depending on the service plan used to create the instance of the Job Scheduling service.

Account	Service Plan	Authentication Type
trial	lite	Basic Auth
prod	standard	OAuth 2.0

Authentication in Trial

What does the environment look like?

{
  "VCAP_SERVICES": {
    "jobscheduler": [
    {
      "credentials": {
        "user": "sbss_gkokz3ayc90hyyyvqat3chwzvkhzkwrqrvby2zw6pjapxxmecgtrcifezrfzjm294xc=",
        "password": "aa_rr/YIrbkHGVneWSchRfbnFSJjc8=",

To access the values, we can traverse through the JSON:

const VCAP_SERVICES = JSON.parse(process.env.VCAP_SERVICES)
const CREDENTIALS = VCAP_SERVICES.jobscheduler[0].credentials
const USER = CREDENTIALS.user
const PWD = CREDENTIALS.password

Note:
You can use a little node module to access the environment variables: @Sisn/xsenv (see here)

2.3.3. How to call the updateStatus endpoint?

Now that we have all the required info, we can go ahead with composing the request to the REST API (see the documentation).

How to build the URL?
This is the template:

{host}/scheduler/jobs/{jobId}/schedules/{scheduleId}/runs/{runId}

Which HTTP Verb?
For updating the status, we use PUT.

Which payload?
As stated in the documentation, the payload has to be a (stringified) JSON object with the mandatory parameter “success” and an optional “message”:

{
  "success": true,
  "message": "Successful finished long running operation"
}

Which code?
Now, to programmatically execute that request to the REST endpoint of the Job Scheduling service, I’ve decided to use the native node module, but you can use any other convenient module of your choice.

The code shows what we learned above:
1. How we access the information from headers.
2. How we compose the authorization from the environment variables.
3. How we compose the URL with the information extracted from the headers.

And at the end we execute the request:

const doUpdateStatus = function(headers, success, message){
      jobId = headers['x-sap-job-id']
      scheduleId = headers['x-sap-job-schedule-id']
      runId = headers['x-sap-job-run-id']
      host = headers['x-sap-scheduler-host']

      const data = JSON.stringify({success: success, message: message}) 
      const options = {
         host:  host.replace('https://', ''),
         path:  `/scheduler/jobs/${jobId}/schedules/${scheduleId}/runs/${runId}`,
         method: 'PUT',
         headers: {
            Authorization: "Basic " + Buffer.from(USER + ':' + PWD).toString("base64")
         }
      };

      const req = https.request(options, (res) => {
      ...
      });     
      ...   

      req.write(data)
      req.end()   
   })
}

Note:
The above code works only in a SAP BTP Trial account, where the Job Scheduling service instance can only be created with the service plan 'lite'.

Please refer to Appendix 2: Full Sample Code for Trial for the full application code.

2.4. Update Status in Prod, Service Plan 'standard'

In a productive landscape, where the Job Scheduling service instance is created with service plan 'standard', the REST API is protected with OAuth 2.0.
Remember: The code sample above is based on a Trial landscape, where only basic authentication is required.

As such, updating the status takes some more lines of code.
In order to use the REST endpoint, we have to send a valid JWT token.
So we need to obtain a token before we execute the REST request.

2.4.1. Fetch JWT token
We get the token from XSUAA.
The credentials required for that call are again provided by the Job Scheduling service in the environment, but this time we don’t get a user and a password, but instead we get the clientid and clientsecret.
Furthermore we need to know the URL which we have to call in order to get the token.
It is the oauth authorization server URL provided by XSUAA. We get that URL in the environment as well.
For more information about OAuth, see SAP BTP Backend Service: Tutorial [14]: About OAuth Mechanism.
So we can go ahead and create a helper method for fetching the JWT token:

const fetchJwtToken = function() {
   return new Promise ((resolve, reject) => {
      // VCAP variables containing the OAuth credentials
      const UAA = CREDENTIALS.uaa
      const OA_CLIENTID = UAA.clientid; 
      const OA_SECRET = UAA.clientsecret;
      const OA_ENDPOINT = UAA.url;
      const options = {
         host:  OA_ENDPOINT.replace('https://', ''),
         path: '/oauth/token?grant_type=client_credentials&response_type=token',
         headers: {
            Authorization: "Basic " + Buffer.from(OA_CLIENTID + ':' + OA_SECRET).toString("base64")
         }
      }
      https.get(options, res => {
      . . .

The response is a JSON-structured string containing several properties, so we can get the JWT token as follows:

const responseAsJson = JSON.parse(response)
const jwtToken = responseAsJson.access_token

So here it is, the token.
Now we can call the REST API to update the status.

2.4.2. Update status
Calling the REST endpoint to change the status of a job run is all the same like realized above, just one small difference:
The authentication has to be done with OAuth 2.0 instead of basic authentication.
We have to send the JWT token in the “Authorization” header as follows:

headers: {
   Authorization: 'Bearer ' + jwtToken

See Appendix 3: Sample Code for Prod for the full code.

3. Test

Deploy the application to your account in SAP BTP.
Create a job and enter the action URL similar like this:
https://longrun.cfapps....hana.ondemand.com/runjob

Let it run and quickly navigate to view the run log.
You can see that the endpoint has responded and that the Job Scheduling service has set a yellow status to indicate that the endpoint has responded but the real status of the long-running operation is not clear yet.

Note:
If you don't see the yellow status, you have to be quicker.

After our application has invoked the updateStatus-endpoint of the REST API of the Job Scheduling service, you can see that the yellow status in the run log has changed accordingly.

To test a negative result, create a job and enter the following action URL;
https://longrun.cfapps.....hana.ondemand.com/runjob?doFail=true
It will show the expected error in the log (like coded by us):

Note:
But what if the long-running operation NEVER ends?
What if the endpoint fails while trying to set the status?
The Job Scheduling service needs to be aware of that, there must be a fallback.
And yes, in such cases, the Job Scheduling service has (another) built-in timeout for async jobs.

Summary

Any job which runs for more than 15 seconds is considered long-running.
The endpoint of your application has to respond immediately with status 202.
Once the long-running operation has finished, your app has to update the status of the Job Scheduling service.
To update the status, you have to do a REST request with PUT and the required info.
The required info can be found in the headers and the app environment.
Authentication for REST API: basic authentication in 'lite', OAuth in prod.

The configuration of a job in the Job Scheduling service is the same as usual.

Appendix 1: HTTP Status Code 202

For your convenience, I've copied the definition from here:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

10.2.3 202 Accepted
The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.

The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persists until the process is completed. The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.

Appendix 2: Full Sample Code for Trial

This project can be deployed to a Trial account, it works with the Job Scheduling service instance created with the service plan 'lite'.

manifest.yml

---
applications:
- name: longrun
  host: longrun
  memory: 128M
  buildpacks:
    - nodejs_buildpack
  services:
    - xsuaainstance
    - jobschedulerinstance

package.json

{
  "name": "myapp",
  "main": "server.js",
  "dependencies": {
    "express": "^4.16.3"
  }
}

server.js

const express = require('express')
const app = express()
const https = require('https');

// access credentials from environment variable

const VCAP_SERVICES = JSON.parse(process.env.VCAP_SERVICES);
const CREDENTIALS = VCAP_SERVICES.jobscheduler[0].credentials;
const USER = CREDENTIALS.user; 
const PWD = CREDENTIALS.password;

// our endpoint which is called by Jobscheduler 

app.get('/runjob', function(req, res){       
   // always return success status for async job
   res.status(202).send('Accepted async job, but long-running operation still running.');
   // afterwards the actual processing
   handleAsyncJob(req.headers, req.query.doFail)
});

// our server

app.listen(process.env.PORT || 3000, ()=>{})

// helper

const handleAsyncJob = function (headers, doFail) {
   doLongRunningOperation(doFail)
   .then((result) => {
      doUpdateStatus(headers, true, result.message)
   })
   .catch((error) => {
      doUpdateStatus(headers, false, error.message)
      .then(()=>{
         console.log("Successfully called REST api of Jobscheduler")
      }).catch((error)=>{
         console.log('Error occurred while calling REST api of Jobscheduler ' + error)
      })
   })
}

// our backend operation

const doLongRunningOperation = function (doFail) {
   return new Promise((resolve, reject)=>{
      const letItFail = doFail
      setTimeout(() => {     
         if(letItFail === 'true'){
            reject({message: 'Backend operation failed with error code 123'});
         }else{
            resolve({message: 'Successfully finished long-running backend operation'})   
         }
      }, 3000); // for testing purpose, 3 seconds are long-running enough
   })
}

// our helper method to set the status in Jobscheduler

const doUpdateStatus = function(headers, success, message){
   return new Promise((resolve, reject) => {
      jobId = headers['x-sap-job-id']
      scheduleId = headers['x-sap-job-schedule-id']
      runId = headers['x-sap-job-run-id']
      host = headers['x-sap-scheduler-host']
   
      const data = JSON.stringify({success: success, message: message}) 
      const options = {
         host:  host.replace('https://', ''),
         path:  `/scheduler/jobs/${jobId}/schedules/${scheduleId}/runs/${runId}`,
         method: 'PUT',
         headers: {
            'Content-Type': 'application/json',
            'Content-Length': data.length,
            Authorization: "Basic " + Buffer.from(USER + ':' + PWD).toString("base64")
         }
      };
      
      const req = https.request(options, (res) => {
         res.setEncoding('utf8')
         if (res.statusCode !== 200 && res.statusCode !== 201) {
           return reject(new Error(`Failed to update status of job ${jobId}`))
         }
   
         res.on('data', () => {
            resolve()
         })
      });
      
      req.on('error', (error) => {
         return reject({error: error})
      });
   
      req.write(data)
      req.end()   
   })
}

Appendix 3: Sample Code for Prod

Use this JavaScript file for scenarios in a productive account, where the Job Scheduling service instance is created with service plan 'standard'.
The difference from the previous app is only in the code:
Calling the REST API with an OAuth flow.

server.js

const express = require('express')
const app = express()
const https = require('https');

// access credentials from environment variable (alternatively use xsenv)

const VCAP_SERVICES = JSON.parse(process.env.VCAP_SERVICES)
const CREDENTIALS = VCAP_SERVICES.jobscheduler[0].credentials

//oauth

const UAA = CREDENTIALS.uaa
const OA_CLIENTID = UAA.clientid; 
const OA_SECRET = UAA.clientsecret;
const OA_ENDPOINT = UAA.url;

// our endpoint which is called by Jobscheduler 

app.get('/runjob', function(req, res){       
   // always return success status for async job
   res.status(202).send('Accepted async job, but long-running operation still running.');
   // afterwards the actual processing
   handleAsyncJob(req.headers, req.query.doFail)
});

// our server

app.listen(process.env.PORT || 3000, ()=>{})

// helper

const handleAsyncJob = function (headers, doFail) { 
   doLongRunningOperation(doFail)
   .then((result) => {
      doUpdateStatus(headers, true, result.message)
   })
   .catch((error) => {
      doUpdateStatus(headers, false, error.message)
      .then(()=>{
         console.log("Successfully called REST api of Jobscheduler")
      }).catch((error)=>{
         console.log('Error occurred while calling REST api of Jobscheduler ' + error)
      })
   })
}

// our backend operation

const doLongRunningOperation = function (doFail) {
   return new Promise((resolve, reject)=>{
      const letItFail = doFail
      setTimeout(() => {     
         if(letItFail === 'true'){
            reject({message: 'Backend operation failed with error code 123'});
         }else{
            resolve({message: 'Successfully finished long-running backend operation'})   
         }
      }, 3000); // for testing purpose, 3 seconds are long-running enough
   })
}

// jwt token required for calling REST api

const fetchJwtToken = function() {
   return new Promise ((resolve, reject) => {
      const options = {
         host:  OA_ENDPOINT.replace('https://', ''),
         path: '/oauth/token?grant_type=client_credentials&response_type=token',
         headers: {
            Authorization: "Basic " + Buffer.from(OA_CLIENTID + ':' + OA_SECRET).toString("base64")
         }
      }
      https.get(options, res => {
         res.setEncoding('utf8')
         let response = ''
         res.on('data', chunk => {
           response += chunk
         })
         res.on('end', () => {
            try {
               const responseAsJson = JSON.parse(response)
               const jwtToken = responseAsJson.access_token            
               if (!jwtToken) {
                  return reject(new Error('Error while fetching JWT token'))
               }
               resolve(jwtToken)
            } catch (error) {
               return reject(new Error('Error while fetching JWT token'))               
            }
         })
      })
      .on("error", (error) => {
         console.log("Error: " + error.message);
         return reject({error: error})
      });
   })   
}

// our helper method to set the status in Jobscheduler

const doUpdateStatus = function(headers, success, message){
   return new Promise((resolve, reject) => {
      return fetchJwtToken()
         .then((jwtToken) => {
            const jobId = headers['x-sap-job-id']
            const scheduleId = headers['x-sap-job-schedule-id']
            const runId = headers['x-sap-job-run-id']
            const host = headers['x-sap-scheduler-host']
         
            const data = JSON.stringify({success: success, message: message})             
            const options = {
               host:  host.replace('https://', ''),
               path:  `/scheduler/jobs/${jobId}/schedules/${scheduleId}/runs/${runId}`,
               method: 'PUT',
               headers: {
                  'Content-Type': 'application/json',
                  'Content-Length': data.length,
                  Authorization: 'Bearer ' + jwtToken
               }
            }
            
            const req = https.request(options, (res) => {
               res.setEncoding('utf8')
               const status = res.statusCode 
               if (status !== 200 && status !== 201) {
                  return reject(new Error(`Failed to update status of job ${jobId}. Error: ${status} - ${res.statusMessage}`))
               }
         
               res.on('data', () => {
                  resolve()
               })
            });
            
            req.on('error', (error) => {
               return reject({error: error})
            });
         
            req.write(data)
            req.end()   
      })
      .catch((error) => {
         console.log('ERROR: failed to fetch JWT token: ' + error)
         reject(error)
      })
   })
}