Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Showing results for 
Search instead for 
Did you mean: 
Beyhan
Product and Topic Expert
Product and Topic Expert
35,951

Overview


SAP BTP, Cloud Foundry runtime uses stacks, a prebuilt root file system that in combination with application code and buildpacks is the base to build the application container. The stack in the SAP BTP, Cloud Foundry environment is based on a Linux system and is called cflinuxfs<x>.

Stacks receive regular updates to address security issues and vulnerabilities, which your applications can pick up through new releases and version updates of the SAP BTP, Cloud Foundry Runtime. To get information about version updates and other Cloud Foundry related topics, see What’s New for SAP Business Technology Platform.

Currently in the SAP BTP, Cloud Foundry environment, all applications run on a stack called cflinuxfs3, which is based on Ubuntu Bionic Beaver 18.04. This stack was deprecated in December 2022 and is planned to be removed from the SAP BTP, Cloud Foundry environment in May 2023 after support for Ubuntu Bionic Beaver 18.04 has ended.

The successor stack cflinuxfs4, based on Ubuntu Jammy Jellyfish 22.04, has been available on an opt-in basis since December 2022 and is planned to become the default stack in March 2023.

This change affects anyone who is running buildpack-based applications in the SAP BTP, Cloud Foundry environment.

To help you understand what is going to happen, we’ve created a simplified timeline of the planned changes:


Cloud Foundry Stack Migration Timeline


 

What Happens When cflinuxfs4 Becomes the Default Stack?


Once cflinuxfs4 is designated as the default stack, all new applications will use this stack automatically. Existing applications running in the SAP BTP, Cloud Foundry environment won’t be affected by this change, but it is recommended that you update them to use the new stack by following the guide below, as the old stack will be removed in May 2023 and your app can stop working in case it is not compatible with the cflinuxfs4 stack.

To start using cflinuxfs4 with existing applications, you can push the application and designate the stack manually using:
cf push <APP_NAME> -s cflinuxfs4

If you’re using the blue-green deployment method to push your applications, they are automatically deployed on cflinuxfs4 when it is declared the default stack. This only works if you delete the unused blue or green applications. For more information on blue-green deployment, see https://docs.cloudfoundry.org/devguide/deploy-apps/blue-green.html.

You can continue to use cflinuxfs3 for a transitionary period. This means you can also manually switch back to cflinuxfs3 if you notice problems using the new stack by using the “-s” command when pushing your application:
cf push <APP_NAME> -s cflinuxfs3

cflinuxfs3 will be permanently deleted in May 2023. Afterwards it is no longer possible to manually switch back to cflinuxfs3. For more information on stacks, see https://docs.cloudfoundry.org/devguide/deploy-apps/stacks.html#available-stacks.

Risks and Consequences


If the applications were running, the restart causes a short downtime. However, if your applications are incompatible with the cflinuxfs4 stack, it is possible that they stop working and cannot be restarted.

To maintain functionality and compliance with security standards in the SAP BTP, Cloud Foundry environment, you must migrate your applications to cflinuxfs4. We recommend that you start testing your applications with the new stack as soon as possible to be able to fix potential issues. Only applications that have been migrated to cflinuxfs4 before the force-migration will not suffer from downtimes.

To assist you, we have compiled a short guide that walks you through the required migration steps.

Migrate to the cflinuxfs4 Stack with the CF CLI


Prerequisite:


Make sure you are using a recent version of the Cloud Foundry command line interface (CF CLI). We recommend using version 8. For information about the newest version, see https://github.com/cloudfoundry/cli/releases.

1. Determine Your Current Stack


If you are unsure which stack your applications are currently running on, you can determine it by using the CF CLI. Run the command:
cf app <APP_NAME>

The resulting output displays information about the application you specified. The output also includes a line starting with “stack:” followed by the name of the stack currently in use.

If cflinuxfs3 is shown here, your application is still using the deprecated stack. If cflinuxfs4 is listed, your application is already using the new stack and no further action is required.

The CF CLI supports the use of third-party plugins. To check which stacks your applications are using, you can use the Stack Auditor CLI Plugin to list applications for each org that you have access to. To see all the applications in your deployment, ensure that you are logged in to the CF CLI as a user who can access all orgs. We do not recommend using the ‘change-stack’ command provided by Stack Auditor plugin. While it can change your stack, it does not provide logs or any information about what went wrong if the operation fails. It also does not work on ARM Macs and uses the deprecated Cloud Foundry API v2. However it works well if you want to find out which of your applications require migration. For more information, see https://docs.cloudfoundry.org/adminguide/stack-auditor.html. Note that this plugin is not supported by SAP.

Alternatively, you can use the jq tool and the CF CLI to query the CF APIs for this information. Note that the jq tool is also not supported by SAP. For more information, see https://stedolan.github.io/jq/.

Using the jq tool, you can run the following script to get an overview of the stack and buildpacks used for all applications within a Cloud Foundry org:
cf curl "/v3/apps?per_page=5000&include=space.organization" | jq '(.included.spaces | INDEX(.guid)) as $spaces | (.included.organizations | INDEX(.guid)) as $orgs | [ .resources[] | {app: .name, org:$orgs[$spaces[.relationships.space.data.guid].relationships.organization.data.guid].name ,space: $spaces[.relationships.space.data.guid].name , lifecycle} ]'

If you want to filter for cflinuxfs3 to display the applications that require manual migration, use:
cf curl "/v3/apps?per_page=5000&include=space.organization" | jq '(.included.spaces | INDEX(.guid)) as $spaces | (.included.organizations | INDEX(.guid)) as $orgs | [ .resources[] | select(.lifecycle.data.stack == "cflinuxfs3") | {app: .name, org:$orgs[$spaces[.relationships.space.data.guid].relationships.organization.data.guid].name ,space: $spaces[.relationships.space.data.guid].name , lifecycle} ]' 

 

2. Determine if Buildpacks Used by Your Applications Support the New Stack


With the CF CLI you can check for the available buildpacks and their stack support by running:
cf buildpacks

Buildpacks supporting cflinuxfs4 will list it in the stack column of the output. The SAP Java Buildpack does not list any stack in the output, but it does support cflinuxfs4.

If you’ve designated a specific buildpack version to be used, it may be necessary to update the buildpack version in the manifest.yml of your application. See https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#buildpacks.

For more information on individual buildpacks, see:

 

3. Deploy an Instance of Your Application Using the New Stack


To check that your application is working properly on the new stack, we recommend that you test it before proceeding with the migration. To do so, push your application a separate time using a new name and a separate route by running:
cf push <APP_NAME> -s cflinuxfs4

If your applications use or contain any precompiled binaries, it is possible that you must recompile them. An example of this are applications depending on binary libraries such as OpenSSL or Python, where cflinuxfs4 relies on newer versions than the ones provided by cflinuxfs3.

If everything worked well, you can continue with your productive apps and change their stack by simply defining it in the cf push command or in the Cloud Foundry manifest. To avoid an unwanted reversion to cflinuxfs3, ensure that your Continuous Deployment automation deploys your applications with cflinuxfs4 as well.

Note that after your application has been deployed using the new stack cflinuxfs4 once, your selection will be saved for future cf push operations. This means that whenever you push the application afterwards, the SAP BTP, Cloud Foundry environment will automatically select the new stack as destination.

Migrate the cflinuxfs4 Stack Using the SAP BTP Cockpit


Instead of using the CF CLI, you can switch from cflinuxfs3 to cflinuxfs4 and vice versa in the SAP BTP cockpit.

Prerequisites:


You need to have the Space Developer role assigned to you.

1. Determine Your Current Stack


You can find your current stack listed on the Overview page of your application in the SAP BTP cockpit. Here, the current stack is cflinuxfs3:




2. Determine if Buildpacks Used by Your Applications Support the New Stack


If the Change Stack button appears on the Overview page of your application, this means that the option to switch between cflinuxfs3 and cflinuxfs4 is available.

If you deploy a new application, it should use the new cflinuxfs4 stack by default. Just to make sure that’s the case, you can always check this information on the Overview page of your application.

3. Changing the Stack


To change your current stack, perform the following steps:

  1. Navigate to the Overview page of your buildpack-based application.



For optimal performance, we recommend that you turn off temporarily the automatic refresh from the upper right-hand corner of the page. You can turn it back on after you change the stack.




  1. Choose Change Stack.



  1. Choose a stack from the drop-down list.


 


The currently selected stack in this example is cflinuxfs3. If you want to migrate to cflinuxfs4, select it from the drop-down list.


If you haven’t turned off the automatic refresh yet, please do so now.




  1. Please read the instructions carefully. Then, check all the boxes.



  1. Choose Save. Applying the change may take some time.


74 Comments
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Sourabh,

 

In general it is fine to change the stack via SAP BTP Cockpit or via the  mta.ymal. One of the two should be enough. I don't see that you missed any step. Regarding the error it is hard for me to get an understanding what happened from the provided logs. Changing the stack in the SAP BTP Cockpit won't touch your service bindings and also, adding only the "stack" parameter to your mta.yaml shouldn't do that. I would suggest you to check out the MTA troubleshooting guid and open a support ticket if it doesn't help.

 

Best regards,

Beyhan
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hello Andreas,

 

From the logs you provided I think that the error you get is not related to the CF stack. It looks like CF is not able to configure a start command for your app. I would suggest you to check how CF determines the start command for an app here. Additionally, please check the docs for the command attribute in the app manifest.

Best regards,

Beyhan

tobias_haak
Advisor
Advisor
0 Kudos
Hi,

we have 2 applications running in the same CF space. In the BTP cockpit for one the "Change Stack" button was displayed and I do the migration. Why does the button not show for the second application? Does that mean, that we MUST go via the CLI approach?

Thanks and regards,

Tobias
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Tobias,

the logic of the "Change Stack" button is as follow. If it is shown but disabled that means you are missing the space developer role. If the button isn't shown at all that means the change via it isn't possible. There are situation where the automatic change isn't possible like no buildpack information is available in the lifecycle metadata of the app because CF automatic buildpack detection is used. In this case I would suggest to go via the CLI approach.

 

Best regards,

Beyhan
SumitKundu
Active Participant
0 Kudos
Hi

Does this affect mobile development kit (MDK) client app deployed in CF too?

Best regards,

Sumit
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Sumit,

 

the MDK client is affected. To my knowledge if you open the Mobile Services admin cockpit with a user with the space developer role the Mobile Services will automatically upgrade to cflinuxfs4. Alternatively you can migrate the stack of the "BindMobileApplicationRoutesToMe" as described in this blog.

Best regards,

Beyhan
wincotiral
Member
0 Kudos
Hi Beyhan,

I tried to migrate several apps (nodejs_buildpack) using the Change Stack button in the BTP Cockpit. Application Information now shows that the stack is now in cflinuxfs4, but the Package Uploaded indicates FAILED.

Tried to do a CF restage in the CLI, but it gets an error "Error staging application: BuildpackCompileFailed - App staging failed in the buildpack compile phase".

 

Best regards,

Win
budh_ram
Associate
Associate
0 Kudos
Hi Beyhan,

 

Is there any exact date decided when SAP will trigger restage of our cf applications?

 

Thanks,

Budh Ram
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Win,

 

What happened after the Change Stack action in the BTP Cockpit? Could the apps start successfully? Most probably the applications are not compatible with the new stack and that is why you see the error during restage. In the mean time we have a SAP note about apps incompatible with the new stack which I will recommend you to check. You will need a BTP user to view it but most probably you have one.

 

Best regards,

Beyhan
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi Budh,

 

to my knowledge the automated update from the platform side to the new cflinuxfs4 stack will start on 18th of May. The update will take several days that is why there is no exact day. The update will restage an application only if the build process with the new stack is successful. As described in the blog applications which are already running on the new stack won't be touched. Additionally, in the mean time there is also an opt-out possibility from the automated update described in this SAP note. I won't recommend to opt-out because the old stack is deprecated, won't get any updates and will be removed at some point. I also won't recommend to wait for the automated update because of the "Risks and Consequences" described in the blog.

Best regards,

Beyhan

ajankow
Discoverer
0 Kudos
Hello,

 

cf restage is also triggered when we use the "change stack" in the BTP cockpit, right?

Best Regards,

Andreas
jkmatthews515
Explorer
0 Kudos
Hi jonathan.bregler ,

I am still facing this issue even after updating all libraries to the latest version in my application.

Error: Connection failed (RTE:[300002] OpenSSL is not available: Unresolved symbol (********-****-****-****-************.hana.prod-us20.hanacloud.ondemand.com:443))

When I revert back to cflinuxfs3 the application resumes working.

When I run locally in BAS, I do not encounter an error.

Are there any other things to try?

Thanks,

Joshua
breglerj
Product and Topic Expert
Product and Topic Expert
Hi jkmatthews515 ,

updating the libraries should be enough. If it doesn't work, there is likely still an older version of a library in your application (maybe the node_modules folder or the package-lock.json file weren't updated). I suggest you open a support ticket to have it analyzed.

BAS doesn't use the CF stack when running locally which is why you don't see the error there.

Best,

Jonathan
jkmatthews515
Explorer
0 Kudos

The resolution to this was to add/update sap-hdbext-promisfied to the latest version.

Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Budh,

I wanted to provide a correction regarding the date mentioned in my previous comment. The automated update from the platform side to the new cflinuxfs4 stack will actually start on the 17th of May, instead of the 18th. I apologize for any confusion caused by the incorrect date.

 

Best regards,

Beyhan
Marc-An_BERTHET
Participant
0 Kudos

Hello,

Thank you for the tips.

 

We did the following action from our side to make it work :

 

Open a new terminal on the project

Update the version of hdbcli in the requirements.txt file :

hdbcli==2.16.21

 

Install the sap-hdbext-promisfied (not sure if it was useful) :

In command line :

npm install -g sap-hdbext-promisfied

 

Deploy the update :
In command line :

cf push <APP_NAME> -s cflinuxfs4
ltomazeli
Product and Topic Expert
Product and Topic Expert
0 Kudos
I was expecting my stack to be updated automatically by now but nothing has changed. I still see the warning message and the force upgrade has not happened. Any news on this?
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Lineker,

the automated migration won't touch an application which fails to stage/build with the new cflinuxfs4 stack or will revert it to the old version if it fails to start with the new cflinuxfs4 stack. The migration marks the issue for an app with a label like "failed-restart" or "failed-staging". You should be able to find the labe with "cf labels app <my-app-name>". As mentioned in the blog post there are different reasons why an application could be incompatible with the new stack. This is a case where the app owner have to take care for the stack update and I can just encourage you to do this because the old cflinxufs3 stack reached its end of life.

 

Best regards,

Beyhan
ltomazeli
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Beyhan,

The weird thing is that we tested cflinuxfs4 on the dev environment and everything works fine. So our expectation was that the other environments were going to update automatically.
Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi Lineker,

 

have you checked about the mentioned labels? Are they present?

 

Best regards,

Beyhan

ltomazeli
Product and Topic Expert
Product and Topic Expert
0 Kudos
I found a fail-restart label. thanks for your help
scatudal
Product and Topic Expert
Product and Topic Expert
0 Kudos

Hi Beyhan,

We changed the stack of each applications that are deployed in the space but still get the warning above the application list. When we navigate to each application details, we see that the stack has been properly migrated: Cloud Foundry Linux-based filesystem (Ubuntu 22.04) (cflinuxfs4)

We observed the same thing in other environments.  Is this a know issue or there's something we still have to do?

Thanks

Beyhan
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Sylvain,

 

this is a generic warning message which informs about the deprecated cflinuxfs3 stack. It doesn't meant that there are apps in the CF environment which are running on that stack. I will provide this feedback to our UI team. Thank you for the feedback.

 

Best regards,

Beyhan
scatudal
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Beyhan,

Thanks for your answer and for bringing it up to the UI team. We are developing a custom solution for a customer who was worried about the warning.

Thanks again,

Sylvain