SAP Developer Challenge - APIs - Task 12 - Use the access token to call the API endpoint

[qmacro]

(Check out the SAP Developer Challenge - APIs blog post for everything you need to know about the challenge to which this task relates!)

Well done for making it to the final task of this SAP Developer Challenge on APIs! You're going to finish on a high, by finally calling the API endpoint in the Core Services for SAP BTP API package. But not without a little diversion in the road on the way there 🙂 Let's get to it!

Background

So, at this point, you have completed steps 1, 2 and 3 in this group of tasks. And after a slight detour into JWTs in the previous task, you're now back on track, on the home straight, ready to complete step 4.

1. create an instance of the SAP Cloud Management Service, with a plan that contains the appropriate scope(s) that you need ✅
2. create a service key based on that instance ✅
3. use the details in the service key to request an access token ✅
4. use the access token thus obtained to authenticate a call to the API endpoint

To complete this task you're going to need to recall bits and pieces from prior tasks:

• the GUID of the directory that you created in Task 7
• the service key data that you obtained in Task 9
• the JSON object containing the access token and related metadata that you requested and received in Task 10

How are you going to use each of these pieces of information?

Well, you'll need the directory GUID to replace the {directoryGUID} placeholder in the actual API endpoint that you're going to be calling (the endpoint detail was also mentioned in Task 9). In other words:

GET /accounts/v1/directories/{directoryGUID}

You'll need information from the service key data to know what the base URL of the Accounts Service API to use, because this /accounts/v1/directories/{directoryGUID} API endpoint belongs to that Accounts Service API, remember? Recall that the service key data looks like this (heavily redacted in the .credentials.uaa section for brevity):

{
  "credentials": {
    "endpoints": {
      "accounts_service_url": "https://accounts-service.cfapps.eu10.hana.ondemand.com",
      "cloud_automation_url": "https://cp-formations.cfapps.eu10.hana.ondemand.com",
      "entitlements_service_url": "https://entitlements-service.cfapps.eu10.hana.ondemand.com",
      "events_service_url": "https://events-service.cfapps.eu10.hana.ondemand.com",
      "external_provider_registry_url": "https://external-provider-registry.cfapps.eu10.hana.ondemand.com",
      "metadata_service_url": "https://metadata-service.cfapps.eu10.hana.ondemand.com",
      "order_processing_url": "https://order-processing.cfapps.eu10.hana.ondemand.com",
      "provisioning_service_url": "https://provisioning-service.cfapps.eu10.hana.ondemand.com",
      "saas_registry_service_url": "https://saas-manager.cfapps.eu10.hana.ondemand.com"
    },
    "grant_type": "user_token",
    "sap.cloud.service": "com.sap.core.commercial.service.central",
    "uaa": {
      "apiurl": "https://api.authentication.eu10.hana.ondemand.com",
      "clientid": "...",
      "clientsecret": "...",
      "...": "..."
    }
  }
}

So you will need the value of the .credentials.endpoints.accounts_service_url property from your service key data.

Finally, you'll need of course the access token you obtained in Task 10, i.e. the value of the access_token property in the JSON object that looks like this:

{
  "access_token": "eyJhbGciOiJSUzI1NiIs...",
  "token_type": "bearer...",
  "id_token": "eyJhbGciOiJSUzI1NiIs...",
  "refresh_token": "e72b61a9a9304dde963e...",
  "expires_in": 43199,
  "scope": "cis-central!b14.glob...",
  "jti": "579fea14a1cf47d7ab9e..."
}

"But wait!" I hear some of you cry. "Task 10 was last week. That's, err, more than 43199 seconds ago, right? What's going to happen?"

Well let's find out!

Attempting a call with an expired access token

Let's assume for this experiment that:

• the GUID of your directory is 57675710-7b16-43ec-b64a-ab14660c1b24
• you have your service key data from Task 9 in a file called sk.json
• your access token data from Task 10 in a file called tokendata.json

Let's also assume that the access token data in tokendata.json was indeed obtained on Friday last week, when Task 10 was published.

Using curl (but we'd see the same effect using any HTTP client, of course), let's see what the actual call to the API endpoint would look like:

curl \
  --verbose \
  --header "Authorization: Bearer $(jq -r .access_token tokendata.json)" \
  --url "$(jq -r .credentials.endpoints.accounts_service_url sk.json)/accounts/v1/directories/57675710-7b16-43ec-b64a-ab14660c1b24"

Invoking this returns something interesting, but not entirely unexpected. Here's part of the verbose output from that curl invocation:

> GET /accounts/v1/directories/57675710-7b16-43ec-b64a-ab14660c1b24 HTTP/2
> Host: accounts-service.cfapps.eu10.hana.ondemand.com
> user-agent: curl/7.74.0
> accept: */*
> authorization: Bearer eyJhbGciOiJSUzI1Ni...
>
< HTTP/2 401
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< date: Sat, 26 Aug 2023 09:30:06 GMT
< expires: 0
< pragma: no-cache
< www-authenticate: Bearer error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Jwt expired at 2023-08-22T20:30:22Z", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-vcap-request-id: 7a7c0c79-f3f7-4b19-651f-6c9b8dd2b013
< x-xss-protection: 1; mode=block
< content-length: 0
< strict-transport-security: max-age=31536000; includeSubDomains; preload;

Ooh! Let's examine the content of that WWW-Authenticate HTTP response header:

error="invalid_token"
error_description="An error occurred while attempting to decode the Jwt: Jwt expired at 2023-08-22T20:30:22Z"
error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"

The error description pretty much gives it to us straight. Our JWT, i.e. the access token, has expired! Now, this example is from my context, where I'd obtained an access token earlier last week. This is why the expiry in this example is on 22 Aug. In fact, let's double check.

Checking when the access token expires

Some of you may have been wondering about the expires_in property in the access token JSON object. This is a simple number of seconds (in this case 43199, to be precise) and represents the lifetime of the token. But how does this relate to actual dates and times? It is of course the lifetime starting from whenever the token was generated, so you may end up calculating the actual expiry date and time by adding those number of seconds on to the exact date and time you obtained the access token. But that can be cumbersome.

You can probably guess that the JWT contains expiry information too. In other words, expiry information is also contained within the access token itself, along with the list of scopes, and other things, that you discovered in the previous task. In fact, not only is the exact expiry date and time in there, but also the date and time when the token was issued. The eagle-eyed amongst you may have spotted the iat and exp properties in the payload part of the JWT in the previous task:

{
  "header": {
    "alg": "...",
    "jku": "https://c2d7b642trial-ga.authentication.eu10.hana.ondemand.com/token_keys",
    "kid": "default-jwt-key-1281344942",
    "typ": "...",
    "jid": "iaVmTleRBCIVnVE7veQ9opMtlHnk+3DvKWWsjpsm542="
  },
  "payload": {
    "...": "...",
    "grant_type": "password",
    "user_id": "965a393a-dc96-422f-87ac-9f3d8bb25142",
    "origin": "sap.default",
    "iat": 1692693022,
    "exp": 1692736222,
    "...": "..."
  },
  "signature": "ZVe_aqyLAyXwToCvG...",
  "input": "eyJhbGciOiJSUzI1NiIsI..."
}

These properties are standard registered claim names, defined in the JSON Web Token (JWT) RFC7519. Specifically, they are:

• the "iat" (Issued At) Claim which conveys when the token was issued
• the "exp" (Expiration Time) Claim which conveys when the token expires

The values of these claims (1692693022 and 1692736222) are UNIX epoch values, i.e. the number of seconds since the UNIX epoch (01 Jan 1970), a standard way to measure time.

Let's examine these a little closer, using the power of the command line and a bit of jq, because why not. Getting the value of the actual access token from the JSON object in tokendata.json file, getting it parsed into its component JWT parts (using the jwt command line tool that we learned about in the previous task), and then taking the exp and iat values from the payload part of the JWT, subtracting one from the other:

jq \
  --raw-output \
  '.access_token' \
  tokendata.json \
  | jwt --output=json \
  | jq '.payload | .exp - .iat'

This emits, rather beautifully:

43200

The --raw-output (which can be shortened to -r) tells jq to emit the raw string, rather than try to always emit valid JSON. So if the value is the string 'hello', then the raw version is hello whereas a valid value as far as JSON is concerned is "hello". Yes, a double-quoted string, all on its own, is syntactically valid JSON. See https://www.json.org/json-en.html for more details.

What about the values themselves? Well, if you're running a standard UNIX style environment with the normal tools (such as in a Dev Space in the SAP Business Application Studio) you can use the standard date command to convert from an epoch value.

Starting almost the same as before, let's first emit the two epoch values:

jq \
  -r \
  '.access_token' \
  tokendata.json \
  | jwt --output=json \
  | jq '.payload | .iat, .exp'

This produces:

1692693022
1692736222

We can then feed those into the date command, using the --date (or -d) parameter to display the date and time denoted by the value that follows it, which will be the epoch time preceded with an @ sign to symbolize "this value is the number of seconds since the epoch":

jq \
  -r \
  '.access_token' \
  tokendata.json \
  | jwt --output=json \
  | jq -r '.payload | "@\(.iat)", "@\(.exp)"' \
  | while read -r epochvalue; do
      date -d "$epochvalue";
  done

This produces:

Tue Aug 22 08:30:22 UTC 2023
Tue Aug 22 20:30:22 UTC 2023

It was last Tuesday morning that I requested and received this access token. And we can see that it expired exactly 12 hours (43200 seconds) later, at Tue Aug 22 20:30:22 UTC 2023. And lo and behold, this is precisely the date and time given in the error description returned in the response where we got an HTTP 401 (UNAUTHORIZED) status code:

error_description="An error occurred while attempting to decode the Jwt: Jwt expired at 2023-08-22T20:30:22Z"

So what are we going to do?

Of course, the sledgehammer approach would be to request another access token. But that's bad practice, because the Resource Owner Password Grant grant type requires the resource owner's credentials, and for the consumer script to hold onto them for such purposes is (or should be) frowned upon, and for the script to re-request them from the resource owner each time is an anti-pattern.

Instead, we can request a new token via the Refresh Token grant type, which "allows clients to continue to have a valid access token without further interaction with the user". Isn't OAuth lovely?

Using the refresh token grant type

So we have everything we need already to request a fresh token. This is what must be supplied in such a call:

• the client's identity (the client ID and secret)
• the grant type, which must be refresh_token
• the actual refresh token itself

The refresh token itself was supplied along with the original access token, in the JSON object returned from the call in Task 10, which we saw briefly earlier in this section:

{
 "access_token": "eyJhbGciOiJSUzI1NiIs...",
 "token_type": "bearer...",
 "id_token": "eyJhbGciOiJSUzI1NiIs...",
 "refresh_token": "e72b61a9a9304dde963e...",
 "expires_in": 43199,
 "scope": "cis-central!b14.glob...",
 "jti": "579fea14a1cf47d7ab9e..."
}

And of course, we still have the client ID and client secret in the service key data (stored in the tokendata.json file).

Where do we send such a call? To the same Authorization Server endpoint as before, i.e. to the /oauth/token endpoint we've used already.

So a token refresh call looks very similar to the previous request when the grant type was "password". Here's the structure:

curl \
  --user '<clientid>:<clientsecret>' \
  --data 'grant_type=refresh_token' \
  --url 'authorizationserver/oauth/token'

Note that the resource owner credentials are conspicuous by their absence here. They are not needed, and should not be required!

Here's an actual call. Values are needed from two places - the service key data and the access token JSON object.

From the service key data, the <clientid>:<clientsecret> construction is achieved with a bit of jq inside a command substitution expansion, joining the .clientid and .clientsecret values (found within the .credentials.uaa value, which is an object) with a colon. And the retrieval of the actual authorization server details is also done in a similar way, taking the value of the .credentials.uaa.url property in the service key data.

And from the access token JSON object, the refresh_token value is taken.

curl \
  --user "$(jq -r '.credentials.uaa|[.clientid,.clientsecret]|join(":")' sk.json)" \
  --data 'grant_type=refresh_token' \
  --data-urlencode "refresh_token=$(jq -r .refresh_token tokendata.json)" \
  --url "$(jq -r .credentials.uaa.url sk.json)/oauth/token" \
  | tee tokendata.json

The venerable UNIX command tee is used here to save the output (the new access token and corresponding metadata in a JSON object) in a file (tokendata.json) as well as letting it pass through to STDOUT so we see it too. It's called tee because it's like using a tee pipe fitting in a plumbing context.

And what do you know? We get a freshly minted access token to use, with 12 more hours on the clock:

{
  "access_token": "eyJhbGciOiJSUzI1Niq2...",
  "token_type": "bearer...",
  "id_token": "eyJhbGciOiJSUzI1NiJS...",
  "refresh_token": "e72b61a9a9304dae263e...",
  "expires_in": 43199,
  "scope": "cis-central!b14.glob...",
  "jti": "579fea14a1cf47d7ab9e..."
}

Nice!

It's useful to know as well that you can refresh your token before the expiry. This gives you a chance to build in a robust token management system into your client, and avoid the risk of falling between the gaps between validity periods.

Using a valid access token

Now that we have a fresh, valid access token, we can complete the journey and make the call to the API endpoint. There's nothing special here, so let's get right to trying it out. In fact, the call is going to be exactly the same as before; the only thing that is different now is that the access token is still valid:

curl \
  --silent \
  --header "Authorization: Bearer $(jq -r .access_token tokendata.json)" \
  --url "$(jq -r .credentials.endpoints.accounts_service_url sk.json)/accounts/v1/directories/57675710-7b16-43ec-b64a-ab14660c1b24"

The --silent parameter here is used to suppress the "progress bar" that curl shows while retrieving a resource.

And the call is successful, emitting ...

Well.

That would be giving the task away, wouldn't it!

Your task

Your task is to ensure you have a valid access token. Ideally, you should work through the process above, using your old (and expired) access token to make a first call to the API endpoint, using the GUID of your directory that you created.

You should see the HTTP 401 status code and look at the value of the WWW-Authenticate header in the HTTP response. You should embrace all that this entails and enjoy matching up the expiration date and time stated in that header, and try to match it up with the value of the exp claim in the payload of the JWT that is your old access token.

Then you should run through the process of requesting a new token, using the Refresh Token grant type explained above.

And with this fresh access token, you should make the call again to the API endpoint, to get the details of the directory that you created way back in Task 7.

Once you have this directory detail, which will be in the form of a JSON object, you should take the value of two of the properties in that detail, join them together with a colon, and send them to the hash service. Then, as always, and as described in Task 0, you should reply to this discussion thread with the hash that's returned.

You need to take the values from these two properties:

• displayName
• directoryType

And don't forget to concatenate them with a colon.

That's it ... you've done it!

1. create an instance of the SAP Cloud Management Service, with a plan that contains the appropriate scope(s) that you need ✅
2. create a service key based on that instance ✅
3. use the details in the service key to request an access token ✅
4. use the access token thus obtained to authenticate a call to the API endpoint ✅

Great work.

Hints and tips

Most of what you'll need has already been covered this time in the narrative within the Background section. And you've all worked so hard with these tasks over the month that you're now well prepared for working with APIs in the SAP universe, dealing with OAuth, endpoints, service key information and more.

Well done!

For discussion

In the directory details that were returned from your successful call to the API endpoint, did you also spot the label information that you added during the directory's creation in Task 7? Where was it? What else did you find interesting about the data returned?

[vladimirs_semikins]

ee9d6b53e6e39debccdbc6a907a3b8d5d0565f602eedfba4f05a0b53d40e0022

[vladimirs_semikins]

I got stuck for a few minutes to refresh my token as I was sending `access_token` and not `refresh_token` 🫣 learned something new! 

As this is a last challenge I would like to say thanks @qmacro for this amazing SAP Developer Challenges! The timing of posting all the challenges was just perfect for me. It was matching with the time when I usually read blog posts and enjoy a quiet morning coffee before kicking off my workday. 👍 I've learned a lot, but not yet fully converted to "terminal-only" user 🙃 (I've used `insomnia.rest` as replacement for curl 🤫 ) so looking forward for more blog posts, challenges, or perhaps we should plan CodeJam in Riga on this topic 🤩

[qmacro]

Thanks @vladimirs_semikins that's made my day.

[salilmehta01]

33b2e24db0788ff8230c0f2bcc1cd38fbbaecd7421a53158904902251f543f09

[ajmaradiaga]

b6f201d6264c7b2c2995e680120d8fac40ab460e242c8c868df76214b23884d6

[ADR]

632c72eacecd9a521024845ebdc188e4d145803a64ca512e160e8c1cde7bbcf6

[ADR]

Let me share an interesting finding:

I shared a different answer earlier. That was not matching with others. When I checked with the directory I created during task 7, I found that I played a little with that directory; activated entitlement and user management at the directory level. This activity changes the directoryType value to a different one. 

I created the directory again under a different trial account and did not make any further changes this time. Now it returns the expected directoryType.

By the way, the label values are retuned twice; under labels and under customProperties.

-Anupam

[qmacro]

This is great! Thanks for sharing. Yes, the directory type changes if you make changes / additions to it. I love the adventurous amongst you folks like this, going a little off-piste, but making sure you can get back on track with the task instructions to continue with everyone else. Because doing that helps you learn more, and when you share that with others too, they benefit from your experiments too.

[ADR]

Thanks a lot DJ for this amazing developer challenge; it helped me to explore multiple topics. Thanks for inspiring us. 

-Anupam 

[qmacro]

You're welcome!

[prachetas]

cf709f95917efdfeb9d962abff8b512d8fed7a6c2b5172e6cc846e785e44b507

[SandipAgarwalla]

e7764169652ca5a4a82620279a94f0073dddf075a4ad39d8deb68a8f40a93762

[SandipAgarwalla]

Is the refresh_token only available for Password Grant type? I am using client credentials as grant type and I do not see the refresh token along with the access token. 

[qmacro]

This is a good question, and asking it, helps us all think about the nature of the Client Credentials grant type, and how it differs from what we've been using in this group of tasks (i.e. the Resource Owner Password Credentials grant type). In fact, it also differs from the Authorization Code grant type in a similar way.

In what way does it differ? Well, as the name of the Client Credentials grant type sort of implies, the credentials needed belong to the client (the script, program, etc) itself.

There is no third party, no human, for example, that is the resource owner, who needs to get involved to lend their credentials (in the case of Resource Owner Password Credentials) or confirm they want to delegate authority, by signing in and causing an authorization code to be issued (in the case of Authorization Code grant type).

And if you think about this when asking yourself why a refresh token is not needed in the case of the Client Credentials grant type, you'll now see why. A refresh token forms part of the reauthentication outside of the loop that involves the resource owner. In the Client Credentials grant type, the client's credentials are all that are needed to get an access token - nothing or no-one else is involved.

So there's no need for a refresh sequence, a client can just request a fresh access token with its own credentials and nothing more.

In fact, if you read the RFC for the the OAuth 2.0 Authorization Framework, RFC 6749, specifically the section on the Client Credentials grant type, you'll see that in the section describing the access token response (which is what you're encountering in your case) you'll see this:

A refresh token SHOULD NOT be included. 

 Hope that explains it a bit!

[SandipAgarwalla]

Thanks DJ

That's a great explanation, and a new learning for me. 

[harsh_itaverma]

fff65e60e747287fcca3741411c2fef13aa436031a2bd1ab2701d85d7049e173

[harsh_itaverma]

Yes, the directory properties were there in custom properties and in label entry in an array [7] too.

Well as this is the last task for this challenge, I would like to thank you @qmacro; it has been a great learning for the past month. In addition to exploring "APIs", got a chance to learn a lot of new things from this challenge (jq, curl, jwt, got a chance to explore the btp cli)

Thanks for curating this challenge for all of us. 🙂

It's been amazing to participate and interact with other folks too. (and cross-checking answers had never been so interesting 😛 )

[qmacro]

Thanks @harsh_itaverma and we appreciated all your interactions this month 💪

[Tomas_Buryanek]

087e21b7b44648db0a1a65137645ef4006ff84b86c5ebff1f760458907122e9d

[Tomas_Buryanek]

Really nice and well prepared challenge! Thank you very much for creating it for us!

I had my hard times with CLI, but eventually my commands worked. But for this last challenge I have confess, that I have used REST client.

I learned & re-learned a lot in this challenge. For example this refresh_token is a new thing for me. Few years ago I have developed one solution which requested always a new OAuth token (before/after expiration of the old one). Maybe that service did not have refresh_token implemented. But next time I will always try to look for it and use it 🙂

[qmacro]

Great to hear, thank you @Tomas_Buryanek !

[UweFetzer_se38]

c16f45e8ed0567a59764cd9910f1a4ef71554653eba74a43cda253465a34c40a

[UweFetzer_se38]

Thank you DJ @qmacro for this wonderfull challenge, learned a lot

[qmacro]

Cheers @UweFetzer_se38 !

[geek]

curl \
  --user "$(jq -r '.credentials.uaa|[.clientid,.clientsecret]|join(":")' sk.json)" \
  --data 'grant_type=refresh_token' \
  --data-urlencode "refresh_token=$(jq -r .refresh_token tokendata.json)" \
  --url "$(jq -r .credentials.uaa.url sk.json)/oauth/token" \

Looks a little off. Double quotes on every line except, 'grant_type=refresh_token'

[qmacro]

See https://qmacro.org/blog/posts/2023/08/25/bash-shell-expansion-inside-double-quotes/ 

[geek]

615f8c66a4977490bb9dc371fa6f303ead1ee59d79fb3ae3569f245261f47218

[ceedee666]

5d51929ea2494cc2959471453f35b6c9a33a8daa421ad4ad62a37b55e5aae848

[ceedee666]

Hi @qmacro 

I really liked this developer challenge. Thanks a lot for creating it. 
Breaking down complex topics in small pieces and explaining them thoroughly is what is missing from most, if not all, of the SAP documentation.
I learned a lot. Including rebuilding my dev container and compiling Neovim in it from the sources as the version in the debian images is to old. 🤣

[qmacro]

Thank you @ceedee666 that means a lot to me.

[emiliocampo]

4fd441cc39a989da8435a5e33a149fb75de7863e203d9833124e94002d5735f0

[emiliocampo]

Hi DJ Adams @qmacro ,

Thnk you so much for creating such an engaging and educational challenge. The tasks were thoughtfully designed and provided a perfect blend of complexity and learning opportunities.

Thank you for the time, effort, and creativity you invested in crafting this challenge. Your work is genuinely appreciated, and I look forward to participating in more challenges and learning opportunities that you may offer in the future.

Emilio Campo

[qmacro]

Hi @emiliocampo that is lovely to hear, and I'm glad you enjoyed it.

[bztoy]

09ec42bf866b102b707a162764a5b1e5686a13f0ccee86f2d263f91181ce9eec

[bztoy]

did you also spot the label information that you added during the directory's creation in Task 7?

---

I found that the label information appeared both in property "labels" and an item of property "customProperties" which is an array.

What else did you find interesting about the data returned?

---

It may be property "consumptionBased" that have boolean false as its value which I don't understand it meaning. 😂

and thank you so much DJ Adams @qmacro for this great community challenge, I think a lot of members above have already mentioned how brilliant it is.

One word that I want to add here is #TheFutureIsTerminal 😎

[qmacro]

Hey @bztoy thank you for sharing your thoughts here and in other tasks too, and thanks for the kind words.

And indeed, # TheFutureIsTerminal 💪

[kasch-code]

0d8326443edc832c8e56952af53235d819919a679252413024f8d72aec820e54

[thomas_jung]

667a2b9e75cd148536b9ac00ff5e3ab73210ecacdc347f0b30475fec6b575c65

[johna69]

92280c0a0ada2890e38b6f0d7c74a35420981b54bde6591aa4d62cbb308436eb