CAP Multitenancy TypeScript application fails to access/create tenants

[nicoj]

Our team has set up a Typescript CAP Application and while adapting the application for multitenancy we are facing an issue which causes the application to crash. We couldn't find the cause of the problem, which is why I'm asking on here. In the following, I'll describe the problem in detail.


What's interesting is, that the app starts fine, when the service implementations, which are written in typescript, are not compiled. For the compilation we use an NPM script (defined in the package.json) called "build:ts" which executes the following command:

find ./gen/srv/srv/ -name *.ts -type f -delete && tsc<br>

This script is called during the build process, as defined in the mta.yaml file. The full mta.yaml and package.json can be seen the end of this post.

build-parameters:
  before-all:
    - builder: custom
      commands:
        - npx -p @sap/cds-dk cds build --production
        - npm run build:ts

Now, when building without compiling the TS service implementations the application starts fine, but obviously the event handlers which are implemented therein don't work. This is expected as uncompiled service implementations are not loaded. The problem is, when compiling the TS files using aforementioned build parameters, the application crashes during startup. First we get a bunch of logs like this:

2023-12-12T10:35:42.588+0000 [APP/PROC/WEB/0] STDOUT [cds] -
2023-12-12T10:35:42.588+0000 [APP/PROC/WEB/0] STDOUT Adding typed method stub for calling custom action 'retry'
2023-12-12T10:35:42.588+0000 [APP/PROC/WEB/0] STDOUT to service impl 'ServiceIntegration'<br><br>[...]<br>
2023-12-12T10:35:42.591+0000 [APP/PROC/WEB/0] STDOUT [cds] -
2023-12-12T10:35:42.591+0000 [APP/PROC/WEB/0] STDOUT Adding typed method stub for calling custom function 'dependencies'
2023-12-12T10:35:42.591+0000 [APP/PROC/WEB/0] STDOUT to service impl 'cds.xt.SaasProvisioningService'

[...]

This part seems to go fine. Afterwards our Job-Service implementation is starting up and then the MTX services are starting up, where the first suspicious log message appears.

2023-12-12T10:35:42.595+0000 [APP/PROC/WEB/0] STDOUT [JobService] - scheduling jobs
2023-12-12T10:35:42.613+0000 [APP/PROC/WEB/0] STDOUT [mtx] - bootstrapping MTX services...
2023-12-12T10:35:42.617+0000 [APP/PROC/WEB/0] STDOUT [mtx] - retrieving { tenants: [ undefined ] } { uncached: [ undefined ] }

As you can see it seems to retrieve undefined tenants. It's not crashing yet.Then we get some more logs like the ones before 'Adding typed method stub [...]' and some others. Then it seems to retrieve the tenant t0.

2023-12-12T10:35:42.874+0000 [APP/PROC/WEB/0] STDOUT [trace] - cds.load model : 21.783ms
2023-12-12T10:35:42.879+0000 [APP/PROC/WEB/0] STDOUT [mtx|t0] - retrieving { tenants: [ 't0' ] } { uncached: [ 't0' ] }<br>

Then some we have the following logs of requests to different endpoints of the service-manager which seem to cause the crash.

2023-12-12T10:35:42.997+0000 [APP/PROC/WEB/0] STDOUT [mtx] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_offerings { params: { fieldQuery: "catalog_name eq 'hana'" } }
2023-12-12T10:35:43.006+0000 [APP/PROC/WEB/0] STDOUT [mtx|t0] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_offerings { params: { fieldQuery: "catalog_name eq 'hana'" } }
2023-12-12T10:35:43.117+0000 [APP/PROC/WEB/0] STDOUT [mtx|t0] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_plans {
2023-12-12T10:35:43.118+0000 [APP/PROC/WEB/0] STDOUT params: {
2023-12-12T10:35:43.118+0000 [APP/PROC/WEB/0] STDOUT fieldQuery: "catalog_name eq 'hdi-shared' and service_offering_id eq '7c994e1c-6694-49c9-84b0-46e94cfb70b3'"
2023-12-12T10:35:43.118+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.118+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.158+0000 [APP/PROC/WEB/0] STDOUT [mtx] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_plans {
2023-12-12T10:35:43.158+0000 [APP/PROC/WEB/0] STDOUT params: {
2023-12-12T10:35:43.158+0000 [APP/PROC/WEB/0] STDOUT fieldQuery: "catalog_name eq 'hdi-shared' and service_offering_id eq '7c994e1c-6694-49c9-84b0-46e94cfb70b3'"
2023-12-12T10:35:43.158+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.158+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.229+0000 [APP/PROC/WEB/0] STDOUT [mtx|t0] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_bindings {
2023-12-12T10:35:43.229+0000 [APP/PROC/WEB/0] STDOUT params: {
2023-12-12T10:35:43.229+0000 [APP/PROC/WEB/0] STDOUT token: undefined,
2023-12-12T10:35:43.229+0000 [APP/PROC/WEB/0] STDOUT labelQuery: "service_plan_id eq 'db5dc984-fcc8-45a3-9648-8c2015c2129f' and tenant_id in ('t0')"
2023-12-12T10:35:43.229+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.229+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.278+0000 [APP/PROC/WEB/0] STDOUT [mtx] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_bindings {
2023-12-12T10:35:43.278+0000 [APP/PROC/WEB/0] STDOUT params: {
2023-12-12T10:35:43.278+0000 [APP/PROC/WEB/0] STDOUT token: undefined,
2023-12-12T10:35:43.278+0000 [APP/PROC/WEB/0] STDOUT labelQuery: "service_plan_id eq 'db5dc984-fcc8-45a3-9648-8c2015c2129f' and tenant_id in ('undefined')"
2023-12-12T10:35:43.278+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.278+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.361+0000 [APP/PROC/WEB/0] STDOUT [mtx|t0] - retrieving { tenants: [ 't0' ] } { uncached: [ 't0' ] }
2023-12-12T10:35:43.362+0000 [APP/PROC/WEB/0] STDOUT [mtx|t0] - GET https://service-manager.cfapps.eu20.hana.ondemand.com/v1/service_bindings {
2023-12-12T10:35:43.362+0000 [APP/PROC/WEB/0] STDOUT params: {
2023-12-12T10:35:43.362+0000 [APP/PROC/WEB/0] STDOUT token: undefined,
2023-12-12T10:35:43.362+0000 [APP/PROC/WEB/0] STDOUT labelQuery: "service_plan_id eq 'db5dc984-fcc8-45a3-9648-8c2015c2129f' and tenant_id in ('t0')"
2023-12-12T10:35:43.362+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.362+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.456+0000 [APP/PROC/WEB/0] STDOUT {"date":"Tue Dec 12 2023 10:35:43 GMT+0000 (Coordinated Universal Time)","error":{"status":404},"exception":true,"level":"error","logger":"sap-cloud-sdk-logger","message":"uncaughtException: Tenant 'undefined' does not exist\nError: Tenant 'undefined' does not exist\n    at Object.get (/home/vcap/app/node_modules/@sap/cds-mtxs/srv/plugins/hana/srv-mgr.js:60:25)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async credentials4 (/home/vcap/app/node_modules/@sap/cds/libx/_runtime/hana/pool.js:26:17)","msg":"Error: Tenant 'undefined' does not exist\n    at Object.get (/home/vcap/app/node_modules/@sap/cds-mtxs/srv/plugins/hana/srv-mgr.js:60:25)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async credentials4 (/home/vcap/app/node_modules/@sap/cds/libx/_runtime/hana/pool.js:26:17)","os":{"loadavg":[3.06,3.51,3.52],"uptime":1010143.57},"process":{"argv":["/home/vcap/deps/0/node/bin/node","/home/vcap/app/node_modules/.bin/cds-serve"],"cwd":"/home/vcap/app","execPath":"/home/vcap/deps/0/node/bin/node","gid":2000,"memoryUsage":{"arrayBuffers":3456579,"external":5919150,"heapTotal":74514432,"heapUsed":60897024,"rss":141623296},"pid":228,"uid":2000,"version":"v18.18.2"},"stack":"Error: Tenant 'undefined' does not exist\n    at Object.get (/home/vcap/app/node_modules/@sap/cds-mtxs/srv/plugins/hana/srv-mgr.js:60:25)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async credentials4 (/home/vcap/app/node_modules/@sap/cds/libx/_runtime/hana/pool.js:26:17)","test":"exception","timestamp":"2023-12-12T10:35:43.455Z","trace":[{"column":25,"file":"/home/vcap/app/node_modules/@sap/cds-mtxs/srv/plugins/hana/srv-mgr.js","function":"Object.get","line":60,"method":"get","native":false},{"column":5,"file":"node:internal/process/task_queues","function":"process.processTicksAndRejections","line":95,"method":"processTicksAndRejections","native":false},{"column":17,"file":"/home/vcap/app/node_modules/@sap/cds/libx/_runtime/hana/pool.js","function":"async credentials4","line":26,"method":null,"native":false}],"written_at":"2023-12-12T10:35:43.455Z","written_ts":1702377343455}

In the last line of the previous code block with logs you can see the stack trace where the error occurs. Just after this the server seems to start up as usual outputting that it has started successfully and the container became healthy, just to crash afterwards, as you can see below.

2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT [pool|t0] - effective pool configuration: {
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT min: 0,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT max: 100,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT testOnBorrow: true,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT fifo: false,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT acquireTimeoutMillis: 1000,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT softIdleTimeoutMillis: 30000,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT idleTimeoutMillis: 30000,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT evictionRunIntervalMillis: 60000,
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT numTestsPerEvictionRun: 34
2023-12-12T10:35:43.567+0000 [APP/PROC/WEB/0] STDOUT }
2023-12-12T10:35:43.728+0000 [APP/PROC/WEB/0] STDOUT [hana|t0] - Elapsed time to create new database connection: 158ms
2023-12-12T10:35:43.729+0000 [APP/PROC/WEB/0] STDOUT [hana|t0] - BEGIN
2023-12-12T10:35:43.730+0000 [APP/PROC/WEB/0] STDOUT [hana|t0] - SELECT * FROM TABLE_COLUMNS WHERE SCHEMA_NAME = ? AND TABLE_NAME = ? [ '***' ]
2023-12-12T10:35:43.748+0000 [APP/PROC/WEB/0] STDOUT [hana|t0] - COMMIT
2023-12-12T10:35:43.756+0000 [APP/PROC/WEB/0] STDOUT [cds] - server listening on { url: 'http://localhost:8080' }
2023-12-12T10:35:43.756+0000 [APP/PROC/WEB/0] STDOUT [cds] - launched at 12/12/2023, 10:35:41 AM, version: 7.4.2, in: 2.346s
2023-12-12T10:35:45.742+0000 [CELL/0] STDOUT Container became healthy
2023-12-12T10:35:46.458+0000 [APP/PROC/WEB/0] STDOUT [cli] - ⚡️ cds serve - process.exit
2023-12-12T10:35:46.479+0000 [APP/PROC/WEB/0] STDERR npm notice
2023-12-12T10:35:46.479+0000 [APP/PROC/WEB/0] STDERR npm notice New major version of npm available! 9.8.1 -> 10.2.5
2023-12-12T10:35:46.479+0000 [APP/PROC/WEB/0] STDERR npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.2.5>
2023-12-12T10:35:46.479+0000 [APP/PROC/WEB/0] STDERR npm notice Run `npm install -g npm@10.2.5` to update!
2023-12-12T10:35:46.479+0000 [APP/PROC/WEB/0] STDERR npm notice
2023-12-12T10:35:46.500+0000 [APP/PROC/WEB/0] STDOUT Exit status 1
2023-12-12T10:35:51.914+0000 [API/44] STDOUT Process has crashed with type: "web"

One of my colleagues also made me aware of the fact that the t0 tenant for the service manager is not created when compiling the TS which could be a problem. If you have any clues that could help us to resolve this issue don't hesitate to give a quick hint even if you're unsure.

Attached are the full mta.yaml(mtayaml.txt) and package.json(packagejson.txt) files in case it can help with finding the root cause of the problem.

[martinfrick]

No too sure about this lazyT0 option which is set to "true" in your case... Maybe that's causing an issue. Did you ever try to remove it from your package.json file? Haven't used it tbh...

[Sergen_Batman]

Hi martinfrick,

we added the lazy T0 option to force the creation of the T0 tenant. Unfortunately also does not work without the option. When deploying the services without the "TSC" build command, the T0 tenant is created successfully..

Best regards
Sergen

[martinfrick]

Okay thanks for giving it a try. Unfortunately, not sure what else might be the problem...

[nicoj]

martinfrick

Thanks for your suggestions thus far. We've found the cause of this issue. We built a 'cronjob' service which on startup checks for the jobs in the db and then schedules them appropriately. Unfortunately, during startup the tenant in the context object is undefined, therefore the application crashes.

Although we now determined the cause, I haven't found a solution yet. I want to try to loop through all the tenants manually and then use cds.spawn with the respective tenant to schedule the jobs. The problem is, that I can't seem to find a way to get a list of the currently subscribed tenants. I tried to add a service implementation function to the provisioning service using service.prepend and therein SELECT the tenant entity, but I get the same error where I need to provide the tenant to perform SELECT on the service.entities.tenant. This is where I'm stuck right now. Any suggestions?

[martinfrick]

Hi nicoj,

Any chance using the SaaS Provisioning Service API to retrieve a list of tenants?

https://api.sap.com/api/APISaasManagerService/path/getApplicationSubscriptions

I am doing a similar thing in my SAP-samples scenarios...

https://github.com/SAP-samples/btp-cap-multitenant-saas/blob/main/docu/4-expert/saas-self-onboarding...

Best,

Martin

[nicoj]

The issue was us trying to SELECT from the DB internally during startup of the services/application. This caused the error shown above, since the context object's tenant is undefined when performing the call to the DB like this.