SAP Cloud Connectivity issues due to Expired Certificate

[pjcools]

A number of customers over the last few months experienced a sudden loss of connectivity in their SAP Cloud Platform landscapes. The customer's environments had already been live for sometime so when receiving a call about them totally losing connectivity was odd. I had not heard of this before even though being experienced in this area. So, with a bit of digging - and of course experiencing it myself I thought to share with the SAP Community. With Productive landscapes now being in operation for a few years I would imagine a lot of customers would start to experience this so hopefully this blog post can help and allow them to be proactive.

***Update 14/01/2020 - New version 2.12.2 shows the subaccount certificate validity when getting close to the end date." Further details below.  

What is the Cause?

The sudden loss of connectivity is due to certificates expiring on SAP Cloud Platform - specifically for the subaccount. Each subaccount (I found out) has a number of certificates, and they can expire! A simple action to renew the certificate must be performed - but it must be performed from the SAP Cloud Connector application. To me, this sort of thing should just be automated however at this stage I don't believe there is any automated functionality to do this. There is functionality to set up email but most of the clients I have worked have still not set this up. Definitely a #FeatureRequestPhil item!

The Error

When this occurs you will see a message appear in the Cloud Connector administration summary page. You will see the Disconnected icon as well as the message "Invalid status of handshake response: 401 Unauthorized".


Figure:1 Invalid status - Unauthorised message

If you check the Alerting page you will also see the detailed messages telling you that the certificate has in fact expired. These ones are more detailed and does tell you what has happened. It does tell you that the tunnel connection is broken and cannot be used. This means that applications will not run at all in that particular subaccount.



Figure:2 Detailed Alerting messages - Certificate expired

On the subaccount dashboard you will also see that in the list of subaccounts that the Status is red.



Figure:3 Subaccount Dashboard - Status is broken due to certificate expiry

Checking Certificates

So, how can you check the validity dates of the certificates in SAP Cloud Platform so that you can be proactive? It is actually quite simple - you need to use console commands to interrogate the keystore, specifically checking the validity on certificates held within the keystore. Before running the commands I will provide some background to assist with the understanding.

To understand the keystores I would recommend you logging into the server where the Cloud Connector resides and using a File Explorer to check the following location for the keystore file path. Here is an example for my trial account but the same applies to any live systems.



Figure:4 Certificate keystore file path for SAP Cloud Platform via SAP Cloud Connector

You can find the keystore using the following path. I have assumed this is located on the C:drive.

c:\SAP\scc20\scc_confg\host name\sub account technical name\scc.jks

The key store file we are checking is scc.jks.

A real example of the above can be viewed in the below screenshots and if you have multiple subaccounts linked to the SAP Cloud Connector you will see a number of subaccount technical name folders in the File Explorer. Each subaccount will have it's own keystore (scc.jks file) so be sure to check each for certificate expiry dates.



Figure:5 Subaccount folders within the SAP Cloud Connector

A specific keytool console command is required and I've found that the easiest way to carry this out is to navigate to the directory that contains the keytool from the SAP JVM. This will just ensure it runs successfully. In my experience I always locate the SAP JVM on the C: drive of the server itself and usually install the SAP Cloud Connector on a separate drive (e.g. 😧 drive). This means when you run the command you need to reference the location of the SCC installation. The screenshot below shows this.

To run the command you need to use the Command prompt. The full command is as follows.

COMMAND = keytool -list -v -keystore D:\SAP\scc20\scc_config\Host Name\Neo subaccount technical name\scc.jks

Note: I have used the 😧 drive above but the drive name needs to be the location where the SAP Cloud Connector application is installed. This is a Windows installation example.

The following information is referenced in the above keytool command including:

• Host name e.g. ap1.hana.ondemand.com


• Neo Sub-Account technical name.

You will then need to run the specific command to list the certificates in the keystore.

keytool -list -v -keystore D:\SAP\scc20\scc_config\ap1.hana.ondemand.com\c34dxxxxx\scc.jks

When you run the command you will need to enter the keystore password as you can see below. At first I had no idea what this would be. I tried my S user password for SAP Cloud Platform and also tried the Cloud Connector S userid password but none of them worked. I had not previously set any password for the keystore so just assumed there was none and this worked. I pressed [Enter] to proceed without a password and this was ok.

So, for the keystore password (unless you have set one) just press [Enter] to proceed. 

The actual example I ran recently is included below.



Figure:6 Results from the keytool keystore command to check the Certificate keystore

As you can see from above, the keystore contains 2 entries. The first one is the CA certificate detailed by the Alias name = ca. This was created on the 24/01/2019. If you check the Validity period you can see this is valid until Mon April 22 2024 - so the good news is we still have a little to go before we need to renew this certificate.

The next one is the actual subaccount certificate. This is the one we are interested in renewing.

The alias name in this case will equal the subaccount technical name.

Figure:7 Subaccount certificate expiry information from the keystore

This was also created on the 24 Jan 2019. As shown above you can see the validity period of the certificate and it expires soon. The certificate is valid until Jan 24th 2020 so it will need to be renewed soon to avoid any connectivity issues.

With this new found information we can now construct a list of each subaccount with the dates that the certificates will expire. This should be added to normal BAU (Business As Usual) maintenance activities in organisations.

Renewing the Certificate

So, now that we know the validity dates we can now plan to renew them. This is carried out in the SAP Cloud Connector. In the Cloud Connector administration page you will see the [Renew Subaccount Certificate] icon up in the top right hand corner.



Figure:8 Subaccount certificate renewal button in SAP Cloud Connector

To renew the certificate click on the [Renew Subaccount Certificate] button. A popup window will be displayed asking for the username and password.



Figure:9 Subaccount credentials pop-up screen

Enter your SAP Cloud Platform username and password. This is your S or P userid for access SAP Cloud Platform global accounts or subaccounts. Click on [OK] to confirm and you will see a "Changes were saved" message.

You will need to reconnect to the subaccount because connectivity was lost due to the certificate expiring. To do this simply click on the [Connect] button as displayed below.



Figure:10 Re-Connect to the Subaccount screen

Once this is carried out the subaccount will come back online and that previous message will disappear. You should now see a positive connector state with a Status of "Connected" shown as displayed below.



 

Figure:11 Connected screen showing "Connected" status

If you now look at the [Alerting] section you will see information about the certificate renewal and the subsequent validity end date. Make sure you note this down!!! 🙂



Figure:12 SAP Cloud Connector Alerting screen

As you can see I had expired certificates in multiple subaccounts so had to renew a number of them in December....hence I worked out pretty quickly how to fix them. 🙂

Crisis averted - with following these activities we should not experience any downtime but definitely suggest keeping a record of all of the expiry dates of certificates from all subaccounts.

Version 2.12.2 - Upgrade

Since posting this I have found out through markus.tolksdorf that version 2.12.2 provides more information on the Subaccount certificate. Specifically displaying whether it is valid or not. So, I thought I would provide some screenshots on exactly what this new version displays.

I upgraded my local version of the Cloud Connector and really great to see the additional fields - check below. To do this you can follow previous blogs I have written. For Windows follow this guide here or for Linux follow this guide here.



Figure:13 Version 2.12.2 Cloud Connector showing Subaccount Certificate field

As you can see above there is a new field called Subaccount Certificate - shown as 1 above. Funnily enough mine had expired. You will see that the same error message is displayed about Invalid status of Handshake. Perfect timing :-). As detailed above, we need to click on the [Refresh subaccount certificate] button shown as 2 above. This will display a pop-up window requesting your User name and password. As also detailed above you will need to enter your SAP Cloud Platform username and password.



Figure:14 Refresh Certificate popup screen

Once this is done the Certificate will be automatically renewed.



Figure:15 Certificate Valid screen

You will also see that the Certificate is now valid. If you check the Alerting section you will also see that the certificate was renewed and you will also see the new Valid to date.



Figure:16 Alerting screen showing new Certificate valid to date

As the connection dropped due to the Subaccount certificate expiring you will also need to re-connect. This was already covered above.

I strongly recommend upgrading your version of the SAP Cloud Connector. By default there will be a date shown of the certificate expiry when is getting close to the renewal date (within 30 days) so definitely worth performing the upgrade.

Lastly, I am definitely no expert in this area and still need to work out what happens with the CA certificate renewal but will provide an update in this blog once I find out. Of course if anyone can supply how this is renewed that would be helpful to add.

As always, thanks for reading!!!

[MarkusTolksdorf]

Hi Phil,

starting with 2.12.2, the Cloud Connector shows the status of the subaccount certificate in the subaccount overview screen. So, in addition to the alerts, you will be able to see the expiration Information there, once it is about to expire.

If you like to see the expiration date earlier simpler than in your blog post, you could simply temporarily set the expiration observation  to e.g. 3000 days. Then a lot of alerts will be generated telling about the expiration dates. Afterwards, you can set the observation setting back to a useful one like the Default 30.

Best regards,
Markus

[former_member620227]

Dear Phil,

Hope you’re doing well.!!

Thanks loads to share this blog with SAP community, this will surely help one and all.

With only newer version(s) of SAP Cloud Connector, we will be able to see the the status of the sub-account certificate in the sub-account overview screen.

For me, when the sub-account certificate suddenly expired, I could observe the error on the SCC UI. But, I was not getting the option to Renew the Sub-account Certificate (although logged in through Administrator account). My SAP Cloud Connector Version is 2.9.0.2

I had to renew the certificate quickly in order to limit the downtime, what I did?

1. I exported entire ACL(Access Control List) belonging to that sub-account.

2. Delete the current sub-account configuration.

3. Configured the sub-account again in SCC.

4. Imported back the ACL and tested for the resource availability.

It was a perfect solution to minimize the downtime, and I could fix this within a minute time. (However, I had invested more than 2 min, to confirm that Renew Sub-Account Certificate option was not appearing).

Could you please guide, if something has been missed out in my case. Why the Renew Certificate Option is not appearing for me. Does it because of lower Cloud Connector Version.

Attaching the screenshot as a reference.

Regards,

Shashank

[pjcools]

Hi markus.tolksdorf

Thankyou for this valuable information and good to see updates in this space. I will upgrade and provide some information in the blog post itself and definitely a great reason for customers to update to the latest version.

Do you know if this covers the CA certificate? Any info on this would be great! 

Many Thanks!

Kind Regards

Phil Cooley

[pjcools]

Thanks shashankraj2100 - seems like a bit of work but glad that you found a way. Definitely need to upgrade your version. I believe the Renew Certificate option was only included in versions above 2.9 and would strongly suggest you update to the latest version.

I wrote some blogs around updating for Windows here and Linux here.

2.12.2 has subaccount certificate information so would definitely upgrade to this version.

Thanks & Kind Regards

Phil Cooley

[MarkusTolksdorf]

Hi Phil.

the validity info about CA and system certificate is shown in Configuration->On-Premise screen.

Best regards,
Markus

[MarkusTolksdorf]

Hi Shashank,

Phil is right: refreshing the subaccount certificate had been introduced with 2.10, which is newer than the version you are using.

Best regards,
Markus

[pjcools]

Hi markus.tolksdorf

Yep, know this. I am talking about the CA certificate from SAP Cloud Platform. PLease refer to Figure 6 above where there is an alias of ca included. This is not the CA certificate in the on-premise area - it is similar to the subaccount certificate and it also has an expiry date. My question is - how does this get renewed as there is no button for this.

Additionally, I have upgraded my local Cloud Connector and my Certificate was in fact expired. So I renewed this and I could only see the new date in the Alerting area as I detailed above. There is no date appearing in the subaccount overview screen unless this is only shown within the Observation configuration settings. Can you please check and advise? Would be good to see a screenshot of what this looks like.

Thanks & Kind Regards

Phil Cooley

[MarkusTolksdorf]

Hi Phil,

The CA certificate of the cloud side is something that is exchanged on cloud side. If this happens, there will be an alert on SCC side as well notifying about the need to renew the subaccount certificate..

The date of the expiry of the subaccount certificate will appear only if it is closer than 30 days. Until Then it is simply shown that it is still valid.

Best regards,
Markus

[pjcools]

Thanks markus.tolksdorf will look out for this. Still would like to see this in action and still doubtful on the renewal of the CA certificate. I have a few customers where this may take place in a few years time so have some time :-).

I will update my blog with the latest version information as I think it will be helpful.

Thanks & Kind Regards

Phil Cooley

[MarkusTolksdorf]

Hi Phil,

the CA certficate will be replaced with the current one, when refreshing the subaccount certificate. As long as it is not replaced by a newer one, you will not notice a difference. If it will be replaced before regular expiry, you will be informed via an alert on Cloud Connector.

Best regards,
Markus

[former_member620227]

Dear Phil,

Yup, infact it was a good learning experience for me. Read your blog on 11th Jan and in my case Sub-account certificate got expired on 12th.?

At first, I was surprised by not getting Renewal option, but then figured out this way of renewal.

Thank you so much, I will certainly upgrade SCC Version.

 

Regards,

Shashank

[former_member620227]

Dear Markus,

Thanks loads for sharing this information. Your comments are commendable.?

 

Regards,

Shashank

[pjcools]

Thanks shashankraj2100 - wow! Just in time!!

Yes, I would upgrade, it is simple to do and you will get the most updated features.

[gregorw]

Hi Markus,

it's great that the visibility improved a bit in 2.12.2. But my wish would be to see the expiration date always. Similar to the screens for the UI, System and CA Certificate.

Best regards
Gregor

 

[MarkusTolksdorf]

Hi Gregor,

we'll think about this

Best regards,
Markus

[WRoeckelein]

Hi phil.cooley ,

thanks for this helpful blog.

Do you know if this new need for regular certifcate renewal means that the information in https://help.sap.com/viewer/cca91383641e40ffbe03bdc78f00f681/Cloud/en-US/f16df12fab9f4fe1b8a4122f0fd... in the prerequisite section "After establishing the Cloud Connector connection, this user is not needed any more since it serves only for initial connection setup. You may revoke the corresponding role assignment then and remove the user from the Members list." is now longer true? I assume the user used for certificate renewal needs the same authorization as for the initial setup?

Regards,

Wolfgang

[pjcools]

Thanks wroeckelein.fum - from my experience with the Cloud Connector I think the SAP help is right. Even though the certificate may need renewal the connectivity is still established right? That is, everything is still connected, it is just the certificate is not valid. So, I would agree once the connection is established you don't really need that initial user anymore, however I have always kept this userid in the members list just in case the entire connectivity is lost. Typically, no one really has access to this anyway but maybe good practice to remove it.

Hope this helps!

Kind Regards

Phil Cooley

[WRoeckelein]

Hi phil.cooley ,

for certificate renewal I need to enter a User/Password, so I would think the user entered needs some authorizations...

Regards,

Wolfgang

[pjcools]

Yes wroeckelein.fum OK, so I think your question is whether or not the Userid you enter here needs to have the Cloud Connector Admin role assigned to it within the subaccount. Interesting question. markus.tolksdorf may know the answer to this?

Thanks

Phil

[MarkusTolksdorf]

Hi,

as long as the certificate is valid and not expired, there is no need to have the user being a member of the SAP CP subaccount. Once expiration approaches and renewal is necessary, a user needs to exist for which it can be checked that a renewal is done by an authorized person. Then, either the same or some other user needs to be added with e.g. Cloud Connector Admin role which contains the permission against which the check is done.

Best regards,
Markus

[WRoeckelein]

Hi markus.tolksdorf ,

thanks for the clarification.

Could you add this to the documentation section cited above (https://help.sap.com/viewer/cca91383641e40ffbe03bdc78f00f681/Cloud/en-US/f16df12fab9f4fe1b8a4122f0fd...)?

Thanks,

Wolfgang

[MarkusTolksdorf]

Hi Wolfgang,

it is already mentioned that the user is not needed after as a member after establishing the connection - please check the note in the prerequisites section.

Best regards,
Markus

[mbortolon]

Hello!

What about the shadow connector instance? It still shows the old certificates and I cannot do anything in the shadow portal

[eduardo_jarauta]

Hello Marco or Phil,

when is this synchronized with the shadow instance?

Can the old certificates be displayed if a switch takes place?

Kind regards

[mbortolon]

I had an old version of SCC and I had to copy certificates manually from master to shadow. One week ago I've updated them to the last 2.13.0 version and I have to see if now is automatic

[eduardo_jarauta]

It is explained here:

2941469 - Subaccount Certificate not getting refreshed in Cloud Connector shadow instance - SAP ONE ...

[former_member61388]

I totally agree with gregorw .

There is no point in hiding a valuable info that the system has.

Today I had to do a survey about certificates expiration on our CCs (a lot of CCs) only to find out that I could not do it.

And by the way it would be useful even to clearly display the CC version on some page, if you have to manage a CC you din't install how can you know which version it is?

[MarkusTolksdorf]

Hi Massimo,

with 2.13, the expiration date is shown. The concrete release can be easily seen in the About screen, which is reachable via the top user menu at the top right.

Best regards,

Markus

[former_member61388]

Thank you!
Do exist a upgrade guide to help in the procedure of installing a new CC?

Because you can't have 2 CC instances on the same machine (they both try to use port 8443) and scheduling a total downtime while importing on premise connection configurations, etc sometimes is not easy.

[MarkusTolksdorf]

Hello Massimo,

See https://help.sap.com/viewer/cca91383641e40ffbe03bdc78f00f681/Cloud/en-US/7a7cc373019b4b6eaab39b5ab70...

Best regards,
Markus

[former_member187007]

Hi Phil, thanks for your post.

I am facing the same issue "Invalid status of handshake response: 401 Unauthorized", I have SCC version 2.12.4, my system and ca certificates are up-to-date, everything seems to be ok.

But despite this, I have the following error in Alerting menu option, as you can see there is not such certificate error here like you mentioned in your post but we can see just this error message:

What could be happening here?

Thanks for your help in advanced.

Jhon Jairo.

[former_member187007]

Hi Phil, Now it's working, today I double checked, and everything is fine, maybe it was a temporary issue.

Thanks anyway.

Jhon Jairo.

 

[former_member263546]

Hi Phil, thanks a lot for your great support with this blog.

I have a question regarding the Keytool keystore command used to check the certificate keystore, not sure if is for the recent version of Cloud Connector but the file  scc.jks does not exist, we have scc.p12 instead and the command didn´t work with this file, can you advice, how can we execute the command with  scc.p12?

keytool -list -v -keystore scc.p12

 

Kind Regards

Carmen Donge

[p_harder]

Hi.

it would be nice if all this certificate expiry stuff could somehow be monitored from on-prem Solution Manager. Currently the only thing that is being monitored is up/down status for SCC and the VM it runs in. So much more could be done there. With a hybrid cloud/on-prem landscape there is a need to centrally monitor end-to-end.

best regards,

Pieter Harder

Brabantwater SAP basis

[wagener-mark]

Hi phil.cooley ,

we are using a SAP Fully Activated Appliance (S/4HANA 2020 FSP2), which comes with a Cloud Connector - to test the Ariba Supplier Solution for S/4HANA.

We try to follow this guide: CIG- Cloud connector configuration

During the creation of the subaccount in the Cloud Connector, we get the following error message:

The trace shows the following:

Could you help us with this issue?

Best regards,

Mark

[pjcools]

Thanks mw_overlack - have you checked out the guided answers at all for this issue?

https://ga.support.sap.com/dtp/viewer/index.html#/tree/2183/actions/27936

This is a good resource to go through to see if it contains your particular problem. Another resource could be my other blog post I wrote -> https://blogs.sap.com/2019/01/26/cloud-connector-guided-answers-and-troubleshooting. This has a few ways of troubleshooting. 

I would also check and ensure that the Subaccount user (you are entering) is assigned the Cloud Connector Administrator role within the subaccount you are trying to connect to.

Check that and let me know how you are going with it.

The other possibility is that the firewall does not allow you to connect out of that server (that the SCC is installed on) to the connectivitycertsigning URL and IP addresses. This needs to be allowed.

Thanks

Phil Cooley

[mario_bisonti2]

Yes, I have the same problem.

I would like to check certificate expiratione using keytool, as per note "2816074 - Check the keystore content of a subaccount in SAP Cloud Connector with keytool" but I have only scc.p12 file, not jks

 

Did you solve it?

Thanks a lot

Mario

[todor_petrov]

Hello,

I am probably re-opening the discussion after some time already, the version of the cloud connector is already 2.14.2, but i still cannot find a way to automate the certificate renewal, which would mean - that every month somebody needs to login to the cloud connector and manual refresh the certificates for all subaccounts? Is that really the case or am I missing something?

Thank you,

Todor

[todor_petrov]

Short update - i checked the same on our PROD instance of Cloud connector and there the certificate validity is much longer (e.g. more than an year) compared to the subaccount on Cloud Connector DEV - is this something that can be configured somehow?

[pjcools]

Not every month bimbat - it is normally on a yearly basis. The idea is to set up Alerting to ensure the right staff get wind of the expiry. This does cause problems on alot of customer environments so good to get the word out.

[pjcools]

I don't believe so todor_petrov - there is nothing I have seen to date to extend this. Usually the subaccount certificate has to be extended yearly but newer versions of the SCC do warn you about it so as long as someone is monitoring this you should be fine. Additionally, set up Alerting function to email staff and this will ensure the certificate is renewed before causing issues.