Putting it all together

quovadis · ‎2022 Nov 01

Using istio gateways, gardener dns and kyma dashboard components

This brief is to demonstrate how SAP BTP, Kyma runtime makes it easy to set up an istio gateway with a wildcard public custom domain.

Pre-requisistes:

access to a registrar with a registered public domain you own

access to one of SAP BTP, Kyma runtime-supported DNS providers.

cluster-admin access to SAP BTP, Kyma Runtime (SKR) kubernetes cluster

Putting it all together

For the sake of this brief I have picked two different public domains registrars, namely Google Domains and Gandi.net and two different external DNS providers, namely: Google Cloud DNS and Azure DNS.

When you provision a kyma cluster it gets assigned a secure wildcard public domain name, for instance: .kyma.ondemand.com, which is entirely managed by SAP.

The built-in kyma-system/kyma-gateway is configured with this default domain.

Likewise, your kyma cluster's API server, namely: https://api.<shoot-name>.kyma.ondemand.com, is bound to that domain as well. Thus, a cluster domain cannot be changed once a cluster has been provisioned.

Q. But what if you wanted to have your own domain with a kyma cluster gateway? In other words, how to set up a domain for workloads across a kyma cluster?

A. Here you go. You will need to accomplish these three tasks, namely:

Task A. Bring your own domain (BYOD)

Custom domains are important when it comes to security, identity protection and commercial branding.

Most likely your business already owns registered domains dedicated to cloud applications.

Task B. Set up the domain with an external DNS provider (Google CloudDNS and Azure DNS)

That includes bringing the DNS servers names to your registrar and granting API access to a DNS zone resources to enable DNS provisioning automation on kyma side.

Task C. Configure SAP BTP, Kyma runtime (SKR) cluster DNS resources automation.

SAP BTP,Kyma runtime (SKR) inherits the DNS resources automation from Gardener, albeit this is transparent for SKR users.

The centrally hosted kyma dashboard has a built-in secrets editor that implements a number of presets including the secret presets for all supported external DNS providers.

I shall endeavour to walk you step by step throughout each of these three tasks. However, if you are network-fluent and have complied with the Task A and Task B requirements, you may jump directly to Task C.

Task A. Bring Your Own Domain (BYOD).

The BYOD refresher:

SAP BTP, Kyma Runtime (SKR), with its built-in automation support of major public DNS providers, makes it easy to BYOD a domain to a kyma cluster.

A BYOD domain must have been registered with a domain registrar. You must have access to your domain's DNS settings in the registrar's admin console.

BYOD domains will be made accessible to kyma cluster resources via one of the supported DNS providers. However, you still retain full ownership. liability and responsibility of any of these domains.

1a. Custom domains with Google Domains

Google Domains is a domain registrar. For instance, you may look up your registered domain names at: https://domains.google.com/registrar/

However, Google Domains is not a DNS (domain name service) provider per se. Thus you will need to bring in the DNS servers names from your established external DNS provider.

Steps	Google Domains registered domain's custom DNS settings
Goto DNS settings and switch to Custom name servers. Please make sure the domain, you are about to modify the DNS settings of, is not used for your websites or email. However, If that were the case, you might want to register a different domain dedicated to your kyma workloads
The custom name servers will come from either the Google Cloud DNS or Azure DNS provider. As Google Domains does not support APIs you will have apply them manually.
After saving the new DNS settings these will be effective after a short while.

1b. Custom domains with Gandi.net

Gandi.net is a popular domain registrar and a DNS provider. However, Gandi is not a supported external DNS provider, thus I had to bring in the DNS name servers from an AzureDNS zone manually as well as depicted below:

AzureDNS zone:

Gandi.net domains registrar

Task B. Set up an external DNS provider. Create a DNS zone.

In order to be able to use one of your registered domains with SAP BTP, Kyma runtime, the first thing you need to do with either Google CloudDNS or AzureDNS is to create a DNS zone for your domain and then copy the DNS servers names to your domains registrar.

What is a DNS zone?

A DNS zone is used to host the DNS records for a particular domain. For example, the domain 'quovadis-btp.com' may contain a number of DNS records such as 'mail.quovadis-btp.com' (for a mail server) and 'www.quovadis-btp.com' (for a web site).

Both Google Cloud DNS and Azure DNS service allows you to host your DNS zone and manage your DNS records. A DNS zone provides the name servers that will respond to DNS queries from end users with the DNS records that you create.

With SKR, the DNS records will be managed by the kyma cluster's DNS resources (DNS Provider and DNS Entry, respectively)

1a. Create Google Cloud DNS zone within your GCP project

In order to be able to use one your Google Domains registered domains with SKR you will need to set up a DNS zone to host the DNS records for a chosen domain name with Google Cloud DNS as follows:

Quickstart: Set up DNS records for a domain name with Cloud DNS | Google Cloud Docs

Steps	Google Network Service/Cloud DNS
Goto Network Service/Cloud DNS within your project.
Create a DNS zone for keeping your Google Domains domain's DNS records.
The default SOA and NS records got created.However, additional DNS records for a chosen domain name will be created when configuring the SKR cluster DNS resources. Thus no need to create the records manually.
Copy the following name servers to your registrar domain's custom DNS settings (via a click on the Registrar setup button or via Equivalent REST shown below).

{

  "cloudLoggingConfig": {

    "enableLogging": false

  },

  "creationTime": "<creationTime>",

  "description": "quovadis-btp",

  "dnsName": "quovadis-btp.com.",

  "fingerprint": "<fingerprint>",

  "id": "<id>",

  "location": "global",

  "name": "quovadis-btp",

  "nameServers": [

    "ns-cloud-**.googledomains.com.",

    "ns-cloud-**.googledomains.com.",

    "ns-cloud-**.googledomains.com.",

    "ns-cloud-**.googledomains.com."

  ],

  "visibility": "PUBLIC"

}

1b. Create Azure DNS zone within your Azure subscription resource group.

In order to be able create a DNS zone you need to have a valid Azure subscription and have created a resource group as depicted below:

Azure DNS zone creation steps:

{

    "id": "/subscriptions/<subscriptionId>/resourceGroups/quovadis/providers/Microsoft.Network/dnszones/quovadis-anywhere.com",

    "name": "quovadis-anywhere.com",

    "type": "Microsoft.Network/dnszones",

    "etag": "<etag>",

    "location": "global",

    "tags": {},

    "properties": {

        "maxNumberOfRecordSets": 10000,

        "maxNumberOfRecordsPerRecordSet": null,

        "nameServers": [

            "ns1-**.azure-dns.com.",

            "ns2-**.azure-dns.net.",

            "ns3-**.azure-dns.org.",

            "ns4-**.azure-dns.info."

        ],

        "numberOfRecordSets": 2

    }

}

2. Create an API access to a DNS zone

The kyma cluster will need an application level (API) access to a DNS zone. That's a pre-requisite for enabling the DNS provisioning automation on kyma side.

With Google CloudDNS you will need to create a service account with a service key.

With AzureDNS you will have to create an Azure AD application and service principal that can access AzureDNS zone

2a. Create a GooglePlatform Service Account and a service key

A service key is required for the kyma cluster to get access to the DNS records of the CloudDNS managed domain. The service key must allow for DNS records modifications.

Steps	Google IAM and admin/Service Accounts
In your Google Cloud project goto Google IAM and admin/Service Accounts and create a service account
The service account has been created
Create a service key with the service account credentials. The service key will need to passed to the kyma cluster namespace as a secret

2b. create an Azure AD application and service principal

The main tutorial that explains how to do it (especially if you do not have access to a Windows machine) is here:

https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-port...

For instance, here go AzureDNS API credentials, namely azure.json:

{

  "tenantId": "<tenantId>",

  "subscriptionId": "<subscriptionId>",

  "resourceGroup": "quovadis",

  "aadClientId": "8e4ed817-c1b0-4cab-89a0-************",

  "aadClientSecret": "c91z*************************"

}

and this is how the secret should look like in kyma dashboard:

kind: Secret

apiVersion: v1

metadata:

  name: quovadis-azure-dns-2

  namespace: azure-dns

data:

  clientID: <clientID>

  clientSecret: <clientSecret>

  resourceGroup: cXVvdmFkaXM=

  subscriptionID: <subscriptionID>

  tenantID: <tenantID>

type: Opaque

Good to know:

in case of any error message the error message will be written into the k8s resource; thus the easiest to look it up in plain English is to open a resource in its editor and goto the yaml tab.

Task C. Configure kyma cluster DNS resources

As usual the best is to refer to the DNS providers source documentation from the Gardener project.

The External-DNS-Management project provides examples with more details for DNSProviders (30-provider-<provider-name>.yaml) and credential Secrets (20-secret-<provider-name>.yaml) for all supported provider types.

All the DNS resources are namespaced. Thus, it's probably a good idea to have a dedicated namespace per DNS provider in a kyma cluster.

0. Create a DNS automation namespace

Let's call this namespace: conn-proxy. Goto namespace configuration to set up the following DNS resources.

1. Create an external DNS provider secret

The DNS provider secret is required for DNS resources automation.

There are two ways to create secrets. Either using the kyma dashboard secret presets or with kubectl as shown below:

$ kubectl create secret generic quovadis-btp-dns --namespace=conn-proxy--from-file=serviceaccount.json=<project name>-11111111111.json

secret/quovadis-btp-dns created

2. Create a DNS provider and a DNS entry

DNS Provider:

apiVersion: dns.gardener.cloud/v1alpha1

kind: DNSProvider

metadata:

  name: quovadis-bpt-dns-provider

  namespace: conn-proxy

  labels:

    app.kubernetes.io/name: quovadis-bpt-dns-provider

  annotations:

    dns.gardener.cloud/class: garden

spec:

  domains:

    include:

      - quovadis-btp.com

      - '*.quovadis-btp.com'

    exclude: []

  secretRef:

    name: quovadis-btp-dns

    namespace: conn-proxy

  type: google-clouddns

DNS Entry:

Good to know:

The below UI has changed and it no longer populates the istio ingress gateway as a target choice.

Thus if you want to use the external ip address of the istio ingress gateway as a target you may need to retrieve it using the following command:

kubectl get svc istio-ingressgateway -n istio-system

apiVersion: dns.gardener.cloud/v1alpha1

kind: DNSEntry

metadata:

  name: quovadis-bpt-dns-entry

  namespace: conn-proxy

  labels:

    app.kubernetes.io/name: quovadis-bpt-dns-entry

  annotations:

    dns.gardener.cloud/class: garden

spec:

  dnsName: '*.quovadis-btp.com'

  targets:

    - **.**.**.**

  text: []

  ttl: 600

3. Create an Issuer with ACME (Automated Certificate Management Environment)

The issuer resource is required to sign the certificate for the domain.

apiVersion: cert.gardener.cloud/v1alpha1

kind: Issuer

metadata:

  name: quovadis-btp-ca-issuer

  namespace: conn-proxy

  labels:

    app.kubernetes.io/name: quovadis-btp-ca-issuer

spec:

  requestsPerDayQuota: 0

  acme:

    server: 'https://acme-v02.api.letsencrypt.org/directory'

    email: <email address>

    autoRegistration: true

    privateKeySecretRef:

      name: ''

      namespace: conn-proxy

    domains:

      include:

        - quovadis-btp.com

        - '*.quovadis-btp.com'

4. Create a certificate resource

This has to be done in the istio-system namespace as follows:

apiVersion: cert.gardener.cloud/v1alpha1

kind: Certificate

metadata:

  labels:

    app.kubernetes.io/name: quovadis-btp-cert

  name: quovadis-btp-cert

  namespace: istio-system

spec:

  commonName: '*.quovadis-btp.com'

  issuerRef:

    name: quovadis-btp-ca-issuer

    namespace: conn-proxy

  secretRef:

    name: tls-quovadis-btp-cert

    namespace: istio-system

5. Create an istio gateway resource

We are back to the DNS automation namespace where we shall create an istio gateway for our own wildcard domain. Here goes the resource deployment script:

apiVersion: networking.istio.io/v1beta1

kind: Gateway

metadata:

  labels:

    app.kubernetes.io/name: quovadis-gateway

  name: quovadis-gateway

  namespace: conn-proxy

spec:

  selector:

    istio: ingressgateway

  servers:

    - hosts:

        - '*.quovadis-btp.com'

      port:

        name: quovadis-https

        number: 443

        protocol: HTTPS

      tls:

        credentialName: tls-quovadis-btp-cert

        mode: SIMPLE

    - hosts:

        - '*.quovadis-btp.com'

      port:

        name: quovadis-http

        number: 80

        protocol: HTTP

      tls:

        httpsRedirect: true

6. Create an API rule for a workload using the custom domain

Last but not least we can now decide to direct the traffic through this new gateway. Let's create an API rule for a function that retrieves data from a remote SAP S/4HANA system.

apiVersion: gateway.kyma-project.io/v1alpha1

kind: APIRule

metadata:

  labels:

    app.kubernetes.io/name: quovadis-s4

  name: quovadis-s4

  namespace: conn-proxy

spec:

  gateway: quovadis-gateway.conn-proxy.svc.cluster.local

  rules:

    - accessStrategies:

        - config:

            jwks_urls:

              - >-

                https://<zone>.authentication.<region>.hana.ondemand.com/token_keys

            trusted_issuers:

              - >-

                https://<zone>.authentication.<region>.hana.ondemand.com/oauth/token

          handler: jwt

      methods:

        - PUT

        - POST

        - PATCH

        - HEAD

        - GET

        - DELETE

      path: /.*

  service:

    host: ***.quovadis-btp.com

    name: test-s4

    port: 80

and voila:

Good to know:

The gateway, namely: quovadis-gateway.conn-proxy.svc.cluster.local, can be used from any namespace across your kyma cluster

Conclusion

I hope you have enjoyed reading this blog post.

You have seen how to expose your Kyma cluster workloads with istio gateways over a custom domain.

Whether this has been easy or not I shall leave it to your judgement. The good news is that this is a one-time effort and that a one single wildcard custom domain can be used for any workload across the SKR cluster.

Last but not least, to stay up to date with the latest news on SAP BTP, Kyma runtime make sure to bookmark our Kyma topic page.

Appendix

ACME Directory URL: https://acme-v02.api.letsencrypt.org/directory

{

  "ddLcOs-N9Uo": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",

  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",

  "meta": {

    "caaIdentities": [

      "letsencrypt.org"

    ],

    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",

    "website": "https://letsencrypt.org"

  },

  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",

  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",

  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",

  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"

}

ACME Certificate issuer:

https://docs.bitnami.com/kubernetes/infrastructure/cert-manager/configuration/generate-acme-certific...

https://cert-manager.io/docs/configuration/acme/

https://certbot-dns-google.readthedocs.io/en/stable/

https://smallstep.com/docs/tutorials/acme-protocol-acme-clients

Additional resources

1. A hands-on tutorial leveraging Cloud-fare DNS | SAP Samples

2. New with SAP BTP, Kyma Runtime: Seamlessly connect your website or microservice to a custom domain u...

3. a video tutorial from SAP HANA Academy

https://www.youtube.com/watch?v=AvhDgMFD76s&amp;list=PLkzo92owKnVw3l4fqcLoQalyFi9K4-UdY

4. Beyond the Gardener project there are other public resources that describe the external DNS providers configurations and settings.

5. https://practicaldatascience.co.uk/data-engineering/how-to-create-a-google-service-account-client-se...

6. https://stackoverflow.com/questions/70774865/add-a-service-account-json-to-kubectl-secrets

7. https://stackexchange.github.io/dnscontrol/providers/azure_dns

SAP Community: https://community.sap.com/

SAP Community Topic Page link: https://community.sap.com/topics/kyma

SAP Community Q&A Tags:
Kyma Open Source: https://answers.sap.com/tags/2936b97d-6a90-4cd8-b635-0e51441611eb
SAP BTP, Kyma runtime: https://answers.sap.com/tags/73554900100800003012

Follow me in SAP Community: piotr.tesny

Public custom domains easy with SAP BTP, Kyma runtime.

Putting it all together

Task A. Bring Your Own Domain (BYOD).

1a. Custom domains with Google Domains

1b. Custom domains with Gandi.net

Task B. Set up an external DNS provider. Create a DNS zone.

1a. Create Google Cloud DNS zone within your GCP project

1b. Create Azure DNS zone within your Azure subscription resource group.

2. Create an API access to a DNS zone

2a. Create a GooglePlatform Service Account and a service key

2b. create an Azure AD application and service principal

Task C. Configure kyma cluster DNS resources

0. Create a DNS automation namespace

1. Create an external DNS provider secret

2. Create a DNS provider and a DNS entry

3. Create an Issuer with ACME (Automated Certificate Management Environment)

4. Create a certificate resource

5. Create an istio gateway resource

6. Create an API rule for a workload using the custom domain

Conclusion

Appendix

Additional resources

Get Your SAP HANA Idea Incubator Badge Today!

SCN Mission - SAP HANA Quiz Challenge is now retired

Share your #HANAStory and Win