Advanced Event Mesh:Troubleshooting

CarlosRoggan · ‎2024 Aug 05

This blog post is trying to collect some helpful tips for troubleshooting issues that arise when configuring AEM with REST API.

View logs in AEM Cloud Console

Easiest way is to use the AEM Insights/Datadog capability.
It has to be enabled in your subscription.
The documentation for insights starts here : https://help.pubsub.em.services.cloud.sap/Cloud/Insights/Insights.htm

Make sure to read the “logs” section and the “monitors” chapter.

View logs using CLI

If you don’t have the AEM Insights/Datadog capability, you can use the Solace Command Line Interface (CLI) to view the logs.

What is the Solace CLI ?
The Solace Command Line Interface (CLI) is a text-based interface.
There’s no command line client tool which we would have to download.
Instead, we use the command prompt to connect to Solace via SSH.
Then we can execute the commands.

What is SSH?
SSH stands for "Secure Shell".
It is a network protocol.
It allows to send commands to a remote computer, in safe manner.

There are a few steps we have to go through:
🔹prerequisite: enable Basic Authentication for management
🔹prerequisite: enable SSH (open Port)
🔹find access details
🔹connect via SSH-tool
🔹run commands
🔹optional: disable SSH and basic auth

See below description

Optional: Enable Basic Authentication

This step is only relevant if you disabled basic authentication for switching to OAuth
(e.g. following this blog post or this one).
To re-enable basic authentication, we just have to fire an HTTP request.

Please follow one of the above mentioned blog posts to know more.
The optional step for disabling basic authentication can be found in chapter 5 of this blog.

For today, let’s just quickly summarize the required steps:

1. Get token
In Cloud Console -> “User & Account” -> “Token Management” -> Create
Enable all permissions and copy the generated token.

2. Retrieve Identifier of Event Broker Service
Go to Cloud Console -> Cluster Manager
Open the details of your service
The Identifier can be found in the URL.
In my example:
https://eu10-canary.console.pubsub.em.services.cloud.sap/mc/services/11223344/status
In above URL, the service ID is 11223344

3. Use REST API to enable Basic Authentication
The PUT request:

Request

URL

https://api.solace.cloud/api/v2/missionControl/eventBrokerServices/<id>/sempBasicAuth

Method

PUT

Headers

name: Authorization
value: Bearer eyJhbciOJzI1NsIpZI6...m1Xc

name: content-type
value: application/json

Body

This JSON payload: {"enabled":"true"}

Note:
Make sure to adapt the request to your environment

Note:
To re-disable basic authentication, just re-send the request with adapted payload

Retrieve Access Details

To connect to the Command Line Interface of Solace, we need the URI to connect and we need the credentials to authenticate.
The info can be found in the Cloud Console on the Service Details page.

We open the Cloud Console -> Cluster Manager -> Select Broker
On the “Status” tab, we find the section DMR Cluster
We take a note of the Hostname, we will need to connect to it.
Under “Management Access”, we find the info of the technical users and passwords which we can use to connect.
We take a note of the “viewer” user and password.

Note:
We don’t use the “admin” user, as it doesn’t have permissions to view the logs

In my example, the required info:
🔹Host: mr-connection-p4ir12yrw29.messaging.solace.cloud
🔹User: demovpn-view
🔹PWD: lgdjnukasdsj9c52r2ktc555pg

Enable SSH Access

To connect to AEM via SSH, we need to open the port.
This can be done in the Cloud Console:
Service Details -> “Manage” tab -> “Advanced Options”
Scroll down to “Port Configuration”
Expand “Public Endpoint”
Click “Edit” in the Action Menu on the right side:

On the “Edit Endpoint” dialog, expand the “Management” section and enable the SSH.
We can leave the default port as 22.
Press “Save”.

Now we may connect to AEM via SSH.
The official documentation can be found here:
https://help.pubsub.em.services.cloud.sap/Cloud/enable-cli-for-cloud.htm

Connect via SSH

To connect to the Solace Command Line Interface we can use our preferred SSH client tool (e.g. Putty).
Under Windows or Linux, we can simply use the built-in SSH tool on command line.
Or use GitBash.

As we have already collected the required info, we can compose the connect-command:

<command> <user>@<host> -p <number>

In my example:

ssh demovpn-view@mr-connection-xxxx.messaging.solace.cloud -p 22

Note:
Make sure to adapt above example to your host

Note:
The port number can be omitted if we stick to the default

After running above command in the shell, we need to confirm once with yes:

And enter the password which we stored above.
That’s it, we’re connected to the interface and are ready to run the CLI commands.

Remember:
Don’t use the admin user, as he is not allowed to view logs.

After successful connection, we can try the following commands:

show session

or (adapt your vpn name):

show message-vpn demovpn

or:

show version

See reference for all commands.

View logs

There are several types of logs as described in docu.
Interesting for us: event logs and also system logs.

So now let’s view the logs:

Connect and enter password

ssh demovpn-view@mr-connection-p4ir1234.messaging.solace.cloud

Run command to view the full event log:

show log event

Then scroll down until the end, to view the newest entries

Note:
We can append a filter statement to the command:
For example, to view only errors during runtime with SEMP:

show log event find "Denied SEMP"

Optional: Disable

For the sake of security, we should revert the initial steps:

1. Basic Authentication for Management
This should be disabled, if OAuth is configured.
Just execute the above mentioned PUT call again, but with adapted payload:
use false instead of true

2. Disable SSH port
After the logs have been viewed long enough, the port should be closed again.
Just follow above description and disable the checkbox again.

Recap

To view logs on CLI, we need the following steps:
🔹enable Basic Authentication
🔹Enable SSH (open Port)
🔹Connect via ssh demovpn-view@mr-connection-p4ir12.messaging.solace.cloud
🔹run command show log event

Optional: Sample Log entries

For your convenience, let’s view a couple of log entries, to be expected for common error cases.

1. Error: Parse Access Token

This is relevant when creating an OAuth Profile in AEM.
There’s a setting which tells AEM to parse the incoming JWT token.
AEM fails if the JWT token doesn’t have the expected format.
This occurs when calling AEM with OAuth bearer token (instead of basic auth, or client certificate).
The response of AEM contains an HTTP status 401, even though the JWT token is valid.
In such case, we find the following error in the event log:

SYSTEM: SYSTEM_AUTHENTICATION_SESSION_DENIED:
Denied SEMP session unknown authentication for (null) (admin).
Reason: The "typ" header parameter was not found

Explanation:
In our scenarios, we fetch a valid JWT token from IAS or XSUAA, but in both cases there is not “typ”-claim contained in the JWT.
So AEM throws an error although the JWT is valid

Conclusion:
When creating an OAuth profile, the parseAccessToken-param should be set to false
Note that the default is true, so the param has to be explicitly added with value false.

2. Error: missing OAuth configuration

If we call the SEMP API and forgot to configure OAuth, we get the following error:

SYSTEM: SYSTEM_AUTHENTICATION_SESSION_DENIED:
Denied SEMP session 193.16.224.15 unknown authentication for (null) (admin).
Reason: The token issuer ('iss' claim) did not match the issuer of any active profile

Explanation:
The AEM reads the iss claim in the JWT token and tries to find a suitable OAuth configuration which contains this issuer.
In this case, we forgot to previously create an OAuth profile for this Authorization Server.

Solution:
Create an OAuth profile as described here or there and configure the OAuth client which was used to fetch that JWT, which caused the error.

3. Error: audience not working

When creating an OAuth profile, we can optionally enable the validation of the aud claim.
If set to true, we need to specify the expected value in another setting of the new profile.
If this is not configured properly, we get the following error:
HTTP 401 status with log entry:

SYSTEM: SYSTEM_AUTHENTICATION_SESSION_DENIED:
Denied SEMP session 193.16.224.15 unknown authentication for (null) (admin).
Reason: The "aud" claim did not match the configured required audience

Explanation:
The Authorization Server writes the expected or targeted receiver of the token into the aud claim.
This is always the requesting OAuth client, but there may be more receivers added to the aud claim.
Solution:
To make sure to properly configure this value, we should fetch a JWT token and view its content, to make sure to copy the correct aud value from there.
To view the content of a JWT token we can use the jwt.io website.

4. Positive scenario

If a HTTP request targeting the SEMP API has been successful, we can find the following entry in the event log:

SYSTEM: SYSTEM_AUTHENTICATION_SESSION_OPENED:
SEMP session 3.64.161.174-11176226 internal authentication opened for user admin (admin)

Issues when using REST API to create OAuth Profile

General Status 400 error while executing REST requests

HTTP status 400 are thrown always for any kind of endpoint-dissatisfaction.
But no useful error messages in the response.
What we can do in case of POST requests:
- Check the name and try to remove any special characters
- Check if you’re creating a duplicate, if the entry already exists

Strange issues when creating OAuth Profiles

As of docu, 3 should be fine, as it is the maximum.
However, after creating third profile, subsequent REST calls did fail.
In my case, everything went fine when working with 2 profiles only.

Issues with IAS

A weird issue in IAS

Be careful when copy&pasting attribute names in IAS.
There might be a blank at the end which is not visible.
It seems that the trailing blank is trimmed in the UI, but not trimmed in the backend of IAS.
Hence, the issued JWT contains an attribute with blank - and it doesn’t match the configured property in the OAuth profile.
Leading to an error on login.
This has to be checked by viewing the JWT in the IAS logs.
But need to really thoroughly view the JWT.

Default Attributes in IAS

It makes a difference when creating an "Application" in IAS with the dashboard, or via service instance in Cloud Foundry:
The set of default attributes is different.
As such, make sure to have a look at the attributes section in IAS cockpit.

Checking the issued JWT token

Crucial for troubleshooting is to view the content of the JWT that is sent to AEM.
To do so, IAS offers the possibility to view the issued tokens in the Logs.
The logs are crowded, so you might find below description useful:

Go to IAS-cockpit -> Monitoring & Reporting -> Troubleshooting Logs
Open a different browser and login to AEM Broker Manager, such that a JWT token is issued.

Now, in IAS, we need to adjust few filters, to find the JWT token in the tremendous amount of logs.
First, we adjust the time span to make it short.
Afterwards. press “Go”.
Additionally, in the search filter field we enter “issueJWT”.
Press enter.
We get the log entries that are relevant for us.
IAS also provides the convenience to decode the token for us.
To view the JWT content, we click on “Log Details”.

We might need to make sure that the JWT token corresponds to the desired user, otherwise choose a different entry or extend the time span.

Example Content of JWT token issued by IAS via client-credentials

See below the content of a JWT token in my example:

{
    "sub":"8310b1aa-9a0e-462f-8f47-9504d032fbbe",
    "aud":"8310b1aa-9a0e-462f-8f47-9504d032fbbe",
    "iss":"ias.accounts400.ondemand.com",
    "exp":1719239601,
    "iat":1719236001,
    "jti":"349ae316-741e-4020-a98a-133ad53b897a"
}

Example Content of JWT token issued by IAS via Authorization Code

See below the shortened content of a JWT token in my example:

{
   "first_name":"Johnedit"
   "last_name":"Aemeditor",
   "user_display_name":"Johnedit",
   "member_of_groups":"aemeditorgroup",
   "mail":"johnedit@aem.com",
   "sub":"P000026",
   "user_uuid":"8b60baa9-eeed-411a-bbe5",
   "aud":"8310b1aa-9a0e-462f-8f47",
   "iss":"ias.accounts400.ondemand.com"
}

Note the claims that we defined manually in IAS.
See the other default claims that can be used when configuring the OAuth profile.

XSUAA

When using XSUAA we have to take care with using the discovery endpoint.
In my case, with the current version, the discovery endpoint is working fine - but it doesn't contain a link to the introspection endpoint.
As such, it doesn't work to configure a discovery endpoint in an OAuth profile in AEM.
The solution is to remove the discovery endpoint and use the parameter for introspection endpoint instead.
This is explained in this blog post.
See also the Cloud Foundry UAA API reference for introspect.

Example Content of JWT token issued by XSUAA

See below the shortened content of a JWT token in my example:

{
    "sub":"sb-xsuaaForAEM!t78007",
    "authorities":["uaa.resource"],
    "scope":["uaa.resource"],
    "client_id":"sb-xsuaaForAEM!t78007",
    "iss":"https://subd.authentication.eu12.hana.ondemand.com/oauth/token",
    "aud":["sb-xsuaaForAEM!t78007","xsuaaForAEM!t78007","uaa"],
    "cid":"sb-xsuaaForAEM!t78007",
    "azp":"sb-xsuaaForAEM!t78007",
    "grant_type":"client_credentials",
    . . . 
}

Links

AEM
Enable the CLI https://help.pubsub.em.services.cloud.sap/Cloud/enable-cli-for-cloud.htm

CLI
Solace CLI main: https://docs.solace.com/Admin/Solace-CLI/Using-Solace-CLI.htm
Reference https://docs.solace.com/Admin-Ref/CLI-Reference/VMR_CLI_Commands.html
Quick Reference https://docs.solace.com/Admin/Solace-CLI/cli-quick-reference.htm
Logs: https://docs.solace.com/Monitoring/Displaying-Logs.htm

SAP Cloud Identity Services - Identity Authentication (IAS):
IAS Landing Page
OAuth flows in IAS main page
Authorization Code flow in IAS

XSUAA:
Cloud Foundry UAA API reference for introspect.

Blogs
Configure OAuth for access to Broker Manager.
Configure OAuth for calling SEMP REST API.