Issue with cds-services-bom version >= 2.9.0

[janpur]

Hi colleagues,

some of our CAP apps (all from one github repo, sharing some code, db etc. - see link below) have issue with cds-services-bom version 2.9.0 or newer (incl. the newest 2.10.1 version). 

https://github.wdf.sap.corp/ISUtilitiesIoT/c4e-aiu/tree/master/c4e-aiu-configuration
(in master still the last working version 2.8.2 is used)

When starting the app an error message "Failed to start a new transaction" is displayed on the UI. 

In the logs we see errors that the tenant is unknown (which of course shouldn't be the case, the tenant exists and there were no issues until version 2.8.2...). See log snippet bellow. 

Any ideas what could be wrong here?

Thanks for your help!
Jan

2024-06-20T12:34:37.031Z INFO 7 --- [pool-3-thread-1] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager

at com.sap.cds.framework.spring.config.datasource.SpringTransactionManager.begin(SpringTransactionManager.java:36) ~[cds-framework-spring-boot-2.10.1.jar:na]

at com.sap.cds.services.impl.handlerregistry.HandlerRegistryTools$DescribedHandler.process(HandlerRegistryTools.java:165) ~[cds-services-impl-2.10.1.jar:na]

at com.sap.cloud.mt.subscription.InstanceLifecycleManagerImpl.lambda$getDataSourceInfoInternal$0(InstanceLifecycleManagerImpl.java:82) ~[multi-tenant-subscription-2.10.1.jar:na]

at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:532) ~[spring-tx-6.1.8.jar:6.1.8]
... 197 common frames omitted
2024-06-20T12:34:28.883Z INFO 7 --- [nio-8080-exec-4] c.sap.cds.adapter.odata.v4.BatchAccess : $batch GET /RegisterCodeAssignment?$count=true&$select=HasActiveEntity,HasDraftEntity,ID,IsActiveEntity,readingType_code,registerCode&$filter=(IsActiveEntity%20eq%20false%20or%20SiblingEntity/IsActiveEntity%20eq%20null)&$expand=DraftAdministrativeData($select=DraftUUID,InProcessByUser,LastChangedByUser)&$skip=0&$top=30 500
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction
Caused by: com.sap.cloud.mt.subscription.exceptions.UnknownTenant: Tenant [64e60ebd-d0c9-4c6f-8378-86e849037997] is not known
Caused by: java.sql.SQLException: com.sap.cloud.mt.subscription.exceptions.UnknownTenant: Tenant [64e60ebd-d0c9-4c6f-8378-86e849037997] is not known

at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:405) ~[spring-tx-6.1.8.jar:6.1.8]
... 195 common frames omitted

at com.sap.cds.services.impl.ServiceImpl.dispatch(ServiceImpl.java:221) ~[cds-services-impl-2.10.1.jar:na]

at com.sap.cds.framework.spring.config.datasource.SpringTransactionManager.begin(SpringTransactionManager.java:49) ~[cds-framework-spring-boot-2.10.1.jar:na]

at com.sap.cloud.mt.runtime.TenantAwareDataSource.getConnection(TenantAwareDataSource.java:57) ~[multi-tenant-runtime-2.10.1.jar:na]

at com.sap.cds.services.impl.persistence.JdbcPersistenceService.ensureTransaction(JdbcPersistenceService.java:104) ~[cds-feature-jdbc-2.10.1.jar:na]

at com.sap.cloud.mt.runtime.DataSourceLookup.getDataSourceAndInfo(DataSourceLookup.java:59) ~[multi-tenant-runtime-2.10.1.jar:na]
... 202 common frames omitted

at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:269) ~[spring-jdbc-6.1.8.jar:6.1.8]

at com.sap.cloud.mt.subscription.InstanceLifecycleManagerImpl.getDataSourceInfoInternal(InstanceLifecycleManagerImpl.java:82) ~[multi-tenant-subscription-2.10.1.jar:na]
... 203 common frames omitted

at com.sap.cloud.mt.runtime.TenantAwareDataSource.getDataSourceAndInfo(TenantAwareDataSource.java:46) ~[multi-tenant-runtime-2.10.1.jar:na]
2024-06-20T12:34:28.882Z WARN 7 --- [nio-8080-exec-4] o.s.c.s.ResourceBundleMessageSource : ResourceBundle [i18n/messages] not found for MessageSource: Can't find bundle for base name i18n/messages, locale en

at java.base/java.util.Optional.orElseThrow(Optional.java:403) ~[na:na]
2024-06-20T12:34:28.882Z INFO 7 --- [nio-8080-exec-4] c.s.c.s.i.c.ChangeSetContextImpl : Rollback only status in member PersistenceService$Default marked the ChangeSet 19 as cancelled

at com.sap.cloud.mt.runtime.TenantAwareDataSource.getConnection(TenantAwareDataSource.java:55) ~[multi-tenant-runtime-2.10.1.jar:na]

at com.sap.cloud.mt.subscription.InstanceLifecycleManagerImpl.getDataSourceInfo(InstanceLifecycleManagerImpl.java:76) ~[multi-tenant-subscription-2.10.1.jar:na]

at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:313) ~[spring-jdbc-6.1.8.jar:6.1.8]

[janpur · Accepted solution]

My question was answered in Stack@SAP. This is the reason:
https://github.tools.sap/cap/issues/issues/16019

[BenPatterson · Accepted solution]

I am not able to access these links, can you please share the details of your answer for this? It may help with our issue that has the same error message.