Issue with SAP BTP CredentialStore w/Basic Auth in US10 region?

[SeanKilleen]

Hi all,

I am writing this in addition to the support ticket I submitted because I want to understand if others are seeing this issue.

Background:

• We have been using Credential Store successfully in multiple environments for some time.
• We typically use the credential store via a Kyma secret binding.
• We currently use basic authentication to access the credential store. The username and password are supplied to us via the bindings.

Issue:

• Last night, we began seeing credential store failures in our dev environment (which uses the us10 region of credentialstore, based on its BTP binding).
• The failure response contains a header, "sapcp-credstore-error-code", with the value "invalid_or_missing_credentials".

What I checked:

• I first saw this in my local environment, so I assumed my bindings were out of date. I went to update them, only to realize they're the same as what I saw in the BTP cockpit.
• I confirmed the bindings matched in BTP Kyma and in my local environment, and matched what I saw in the cockpit.
• I reverted to a known good version of the code (based on release tags). The problem persisted.
• I verified that the logic for authentication hadn't changed, that it was being base64 encoded correctly, etc.
• I realized that requests in an another environment, which uses us21, are succeeding.
• Confirmed that the issue is happening for multiple devs in their local environment (e.g. it's not a typo in the bindings I'm using locally somehow).
• Restarted containers in the dev environment where we saw this show up too.

Minimal Reproduction:

• In Postman, I set up a request to the "get all keys" endpoint.
• In the us21 region, with the values copied from that binding, this request succeeds -- I receive back an (encrypted, as expected) response.
• The same call in the us10 region, using username/password values copied from the existing bindings for that environment (which triple-checked to be correct), the request fails with this error.

It appears to be there is an issue with the SAP Credential Store environment and basic auth in the us10 regions.

[SeanKilleen]

I officially found what the problem is I think.

The Kyma binding and the binding in BTP cockpit has an expiration of "2023-12-18T23:47:09".

It seems likely this is the culprit.

So the new question is: why wasn’t that regenerated? Is that a manual process? I would expect that to be kept up to date by BTP and the Kyma bindings rather than me needing to take any action.

[SeanKilleen]

Narrowing it down further, the good news is this may be our fault! Which means I can fix it.

I realized I'd expected the BTP operator within Kyma to be rotating our credentials. So I went to open a GitHub issue there.

Then, I saw https://github.com/SAP/sap-btp-service-operator#credentials-rotation

I missed this part in the docs during initial setup and I assumed that credentials would be rotated by default rather than just expiring and causing a failure.

I'll work through those steps and may have a solution to report back after.

[SeanKilleen · Accepted solution]

I was able to discover the true cause -- thankfully, it was totally my fault! 😄

I thought I had ruled out all of the things that could be caused by our environment, but I missed a very important one, which I'll explain here in case some other poor soul finds themselves in the same predicament. :)

-----

In our Kyma environment, we use the SAP BTP Operator to facilitate our service bindings.

By default, the SAP BTP Operator does not rotate service bindings. It instead allows them to expire and fail. I did not expect this default behavior, and months ago I missed the documentation about enabling credentials rotation.

Once I added the logic to enable the credential rotation (see https://github.com/SAP/sap-btp-service-operator#credentials-rotation), the bindings were immediately rotated and the app began functioning again.