Introducing SAP Kyma runtime dynamic credentials with HCP Terraform

[quovadis]

Terraform stories. SAP Kyma runtime dynamic credentials with HCP Terraform

HCP Terraform is available as a hosted service at https://app.terraform.io. 

It can be used as an OIDC identity provider with your SAP BTP, Kyma runtime kubernetes clusters.

Dynamic Credentials with the Kubernetes and Helm providers

HCP Terraform already supports dynamic credentials with Kubernetes in AWS and GCP.  And I have extended this support to SAP BTP, Kyma runtime clusters.

Let's see how...

Spoiler

https://dev.to/quovadis/terraform-stories-5deh

Let me help explain this comprehensive guide about integrating HCP (HashiCorp Cloud Platform) Terraform with SAP Kyma clusters using dynamic OIDC (OpenID Connect) credentials.

The guide details how to enable Terraform to authenticate and interact with Kyma clusters through OIDC, which is a secure authentication protocol. This setup consists of three main parts:

First, configuring the Kubernetes side involves setting up the OIDC provider in the Kyma cluster. The process uses Gardener's OIDC shoot extension, which allows multiple OIDC providers in a single cluster. The configuration includes creating an OpenIDConnect custom resource that specifies details like the issuer URL (app.terraform.io) and how to map claims to Kubernetes usernames and groups.

The implementation creates RBAC (Role-Based Access Control) identities during the initial cluster configuration. These identities follow a specific format for Terraform Cloud users: organization:<ORG>:project:<PROJECT>:workspace:<WORKSPACE>:run_phase:<plan|apply>. This ensures proper authorization is in place as soon as the cluster is ready.

Second, the HCP Terraform configuration requires setting specific environment variables:
- TFC_KUBERNETES_PROVIDER_AUTH must be set to "true" to enable authentication
- TFC_KUBERNETES_WORKLOAD_IDENTITY_AUDIENCE needs to specify the audience name from the cluster's OIDC configuration

Third, the provider configuration in Terraform needs to handle the dynamic credentials. Terraform provides a kubeconfig token valid for 90 minutes through the tfc_kubernetes_dynamic_credentials variable. This token is then used in the Kubernetes and kubectl provider configurations along with the cluster endpoint URL and CA certificate.

The guide includes several practical examples of retrieving cluster information once the configuration is in place, such as:
- Getting shoot information (domain, Kubernetes version, network configurations)
- Listing availability zones for cluster nodes
- Retrieving the list of installed Kyma modules

Would you like me to elaborate on any particular aspect of this setup, such as the OIDC configuration, RBAC setup, or the provider configuration?

Let me explain the provider configuration in detail, as it's a crucial part of connecting HCP Terraform with your Kyma cluster.

The provider configuration starts with the Terraform block that sets up the basic framework:

```hcl
terraform {
  cloud {
    organization = "<organization>"
    workspaces {
      project = "terraform-stories"
      tags = ["runtime-context"]      
    }
  }
  
  required_providers {
    btp = {
      source = "SAP/btp"
    }    
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
    kubectl = {
      source = "alekc/kubectl"
    }
  }
}
```

This configuration sets up three important providers: the SAP Business Technology Platform (BTP) provider, the official Kubernetes provider, and a kubectl provider for additional Kubernetes operations. The cloud block configures the Terraform Cloud workspace, which is essential for the OIDC authentication flow.

The dynamic credentials configuration is handled through a special variable:

```hcl
variable "tfc_kubernetes_dynamic_credentials" {
  description = "Object containing Kubernetes dynamic credentials configuration"
  type = object({
    default = object({
      token_path = string
    })
    aliases = map(object({
      token_path = string
    }))
  })
}
```

This variable structure allows Terraform to manage the authentication tokens. The `token_path` points to where Terraform can find the temporary authentication token. The aliases map allows you to configure multiple Kubernetes contexts if needed, though in most cases you'll just use the default.

The actual provider configurations use these credentials along with cluster information:

```hcl
provider "kubernetes" {
  host                   = var.cluster-endpoint-url
  cluster_ca_certificate = base64decode(var.cluster-endpoint-ca)
  token                  = file(var.tfc_kubernetes_dynamic_credentials.default.token_path)
}

provider "kubectl" {
  host                   = var.cluster-endpoint-url
  cluster_ca_certificate = base64decode(var.cluster-endpoint-ca)
  token                  = file(var.tfc_kubernetes_dynamic_credentials.default.token_path)
  load_config_file       = false
}
```

The cluster information (host and CA certificate) typically comes from the Kyma cluster's kubeconfig. You can retrieve this information using an HTTP data source:

```hcl
data "http" "kubeconfig" {
  depends_on = [btp_subaccount_environment_instance.kyma]
  url = jsondecode(local.labels)["KubeconfigURL"]

  lifecycle {
    postcondition {
      condition     = can(regex("kind: Config", self.response_body))
      error_message = "Invalid content of downloaded kubeconfig"
    }
    postcondition {
      condition     = contains([200], self.status_code)
      error_message = self.response_body
    }
  } 
}

locals {
  kubeconfig = data.http.kubeconfig.response_body
  cluster_ca_certificate = base64decode(local.kubeconfig.clusters[0].cluster.certificate-authority-data)
  host                   = local.kubeconfig.clusters[0].cluster.server
}
```

This configuration includes several important security and reliability features:
1. The dynamic token is automatically rotated by HCP Terraform (valid for 90 minutes)
2. The configuration includes validation checks for the kubeconfig download
3. The CA certificate is properly decoded from base64 format
4. The kubectl provider has `load_config_file` set to false to ensure it uses only the provided configuration

The combination of these providers gives you full access to both the standard Kubernetes resources (through the kubernetes provider) and custom resources or kubectl commands (through the kubectl provider), all authenticated through the secure OIDC mechanism.

Would you like me to explain more about how these providers interact with the RBAC configuration we discussed earlier, or would you like to know more about any specific aspect of this provider setup?