From Application Logging to Cloud Logging: Service Innovation Guide

[Hariharan-Gandhi]

This service innovation guide explains the innovation journey from SAP Application Logging Service towards SAP Cloud Logging. It shall help customers to understand the benefits and differences to make best possible use of SAP Cloud Logging and to prepare for the migration from SAP Application Logging. It is complemented by a service migration guide describing a procedure to switch from the Application Logging to the Cloud Logging.

Motivation

SAP Application Logging Service successfully served Cloud Foundry application developers for almost 10 years to store, visualize, and analyze their log messages. To improve its usability even further we repeatedly received some key requirements from customers, incl. alerting, configurable retention periods, custom dashboards, and many more. Technical, security, privacy, and compliance restrictions did not allow us to implement these requirements on the underlying multi-tenant stack.

That’s why we created a completely new solution, SAP Cloud Logging. It provides each customer with a dedicated and fully isolated observability stack. This allowed us to not only implement the above-listed requirements but also to innovate by supporting metrics & traces (via OpenTelemetry) and serving major BTP runtimes (Cloud Foundry, Kyma, and others).

SAP Cloud Logging has been proven to be a successful and reliable service, which is why we will deprecate SAP Application Logging Service. This service innovation guide is designed to help you to understand the full potential of SAP Cloud Logging, including its additional benefits, while highlighting the areas where it behaves differently or requires different setup and usage considerations. 

Capabilities & Differences

SAP Cloud Logging comes with many advanced capabilities beyond those of SAP Application Logging Service. The most relevant ones are listed below:

Regarding SAP Cloud ALM check the setup guides for SAP Cloud ALM and SAP Focused Run.

While almost all feature improvements can be simply consumed on top of the existing experience there are few features worth mentioned explicitly:

The authentication system for Cloud Logging has to cover a much more general scenario as it is not only bound to Cloud Foundry. The resulting setup via SAML/IDP requires a one-time setup effort on the customer side compared to the Cloud Foundry-centric setup in Application Logging. We provide detailed documentation on how this can be done. This makes the service future proof to support all SAP runtimes.

As of today, there is one minor feature gap which is the exposure of Cloud Foundry container metrics for Cloud Foundry applications. This gap shall be closed by soon.

Regarding dashboards, the “Statistics” dashboard about quota-based logs dropping is not provided for Cloud Logging Service because it does not drop logs based on quota. The “Metrics” dashboard will be part of the CF container metrics rollout

Service Deployment

Regional Hub

Application Logging is deployed in every Cloud Foundry landscape.

Cloud Logging is a generalized offering that can be jointly used across runtimes, environments, and even data centers. Therefore, it’s deployed once per (compliance) region, from where it’s offered for consumption in all related BTP landscapes and can be commonly used.

 

Cross tenant (CF Space) analytics

Application Logging is a shared stack where authorization is restricted based on tenant (CF Space) access. For a user, who has CF Space Developer role in multiple spaces, this allows “joint log analysis” across all logging service instances from those multiple spaces within that BTP landscape.

As Cloud Logging is built on dedicated stacks for each service instance, it does not offer joint analysis across those service instances. However, as Cloud Logging enables sharing beyond a single space, you could setup single instance to serve apps from multiple spaces, thereby achieving joint analytics. Also note, CLS also offers fine-grained user authorization – so within a single instance you could define different access rule for your users (restrict space-based access to logs, read only, edit dashboard, hide sensitive logs etc.).

Landscape availability

The availability of Cloud Logging in certain BTP landscapes may shortly follow the initial launch of the landscape. Currently, this is the case for BTP in Switzerland (CH20).

Improved Ingestion Security

Securing Ingestion and rotation

Since ingestion to Cloud Logging is flexible and in customers’ control, it is necessary to offer industry standard ingestion authentication. OTel and ingest endpoints support mTLS ingestion which help to comply with most organization’s security standards.

With certificate-based security mechanism users are responsible for timely rotation of certificate to avoid expiration. With just regular re-binding, Cloud logging allows renewing client certificates. Please read the document for best practice on rotation without disruption.

User provided services

We also recommend leveraging User provided service which offers much faster and effective binding in following cases:

• If your deployment does frequent bind/unbind but you do not intend to rotate certificates, then you could create a User provided service based on the Cloud Logging instance’s service key. As the actual bind/unbind is an expensive and time-consuming process involving new certificate generation, this approach w/ UPS makes the binding faster, fail safe and uninterrupted. Also minimizes the number of certificates that needs to be monitored and rotated. When it is actually needed to rotate the certificates, you could create an additional service key, update the UPS and rebind workloads.
• When you are using central CLS instance across different sub-accounts / BTP landscapes, then UPS is way to reuse the existing instance in one location across many.
• Of course, this requires keeping the UPS manually up-to date with the service keys.

Service Plans & Pricing

Application Logging comes with fixed-size service plans charged monthly. In contrast to this, the flexible configuration and auto-scaling of Cloud Logging comes with a more fine-grained approach.

Service Plans

Application Logging service plans are designed based on ingestion quota. Cloud Logging offers service plans along the number of provisioned resources.

Cloud Logging offers

• 2 production-ready service plans (standard & large) which offer net storage capacity from 75 GB to 3.75 TB.
• 1 development plan (dev) which offers all functional features of production plans but lacks cloud qualities, i.e. no data replication, no auto-scaling, and very limited ingestion throughput. è It’s for evaluation purposes only and must not be used in production use cases.
• its development plan also in the context of SAP Build Code

While Application Logging offers a specific lite plan for evaluation purposes at no charge cost there is no comparable offering in Cloud Logging. This is due to the dedicated provisioning of resources per Cloud Logging instance. Furthermore, customers may reduce cost for development scenarios by sharing Cloud Logging service instances across stakeholders and scenarios.

Pricing

In contrast to the fixed-size and full-month approach of Application Logging the pricing approach of Cloud Logging works as follows:

• Metering & pricing is based on provisioned components (foundation, storage, ingestion).
• Autoscaling of underlying components is explicitly captured.
• Metering is done ”hourly”, i.e. is based on what is physically deployed at the metered hour.
• Usage is aggregated into a single metric: “capacity unit”.

The benefits of this approach are metering is highly accurate to actual consumption per hour and selection of service plans incl. specification of upper bounds for auto-scaling components ensures cost planning safety while still allowing for flexible adaption for changes in observability load.

Example: If customers have configured to use autoscaling for ingest (say 10 blocks) and has been using all 10. At a certain hour when there is no ingest, they will scale-in to 2 and be billed for 2 blocks for that hour. (same applies to disk/storage). Note: There is no scale to 0.
So, if users create an instance with min configuration of standard plan (without auto-scaling) - they would’ve consumed 538 Capacity Units after “30 days” (lower for 28 day months and slightly higher for 31 day months).

Activity shorter than a month (mostly in evaluation cases or cases where instances are created & destroyed often): If the instance is created on 5th morning at 9:00 AM à evaluated until 6th evening 16:00 PM and deprovisioned – then the billing is only for the 31 hours instead of the complete month. On the other hand, this is different from Application logging, where it is billed for complete month for each short incarnation of the service’s instances. i.e., each service “recreate” will be fully billed for a month.

Use SAP Cloud Logging cost estimator to determine cost based on your needs.

Eventual billing numbers are reported via a specific service plan, called “overall”, which is used to aggregate billing numbers across all actual service plans (standard, large, dev). Metrics captured on detailed metrics (e.g. number of ingest blocks) can be inspected via Cloud Cockpit.

Sizing

This section discusses how many SAP Cloud Logging instances you need, and how many apps to bind to one instance. We recommend revisiting which apps are bound to which logging service plan instance and not having a one-to-one mapping of service plans. Compared to Application Logging, Cloud Logging instances can handle significantly more log volume. For most use cases, peak ingestion performance should not be a problem for either of the services.

What previously has been sent to multiple service instances can now be sent to one. This solution allows for additional analyses and can be the more cost-efficient option.

For Application Logging, log ingestion is limited by the service plan quota limit, which defines the theoretical maximum of log storage. Cloud Logging instances have auto-scaling capabilities and a pay-per-resource usage model in which it checks usage hourly.

To determine the storage demand, check the statistics dashboard in Application Logging on how many logs (and bytes) you are shipping and how many have been dropped. When calculating the storage demand, consider the data retention. While data retention is 7 days for Application Logging, it is configurable in Cloud Logging Service. An increase in data retention increases your storage demand. For example, increasing retention from 7 to 30 days increases storage demand by the factor of 4.28.

If you have multiple applications forming a single solution and the combined log volume fits a single Cloud Logging instance, we recommend binding them all to that instance. This allows you to analyze your application data together and investigate cross-application problems.

Find more detailed information on sizing, performance, and Cloud Logging Service plans at the Discovery Center.

Instance Planning

While the setup and usage of Application Logging is always confined by a Cloud Foundry landscape Cloud Logging offers way more flexibility for re-using existing instances in multiple contexts. This comes with technical, cost, and legal implications thus requires proper planning.

Reusability / Shareability

Cloud Logging comes with the ability to “share” single instance across various application deployment locations.

• This increases convenience of centralizing observability in a single or fewer instances.
• This drastically reduces cost in contrast to Application Logging where customers need to create different instances per space.

Sharing across spaces:

Sharing across runtimes:

Sharing across DCs in a region:

Sharing across DCs across a region:

• Technically it is also possible to extend the above scenario to share across regions (US <-> EU) depending on the ingest load and latency.
  
  

Tradeoffs

Planning actual instantiation and deployment of Cloud Logging service instances should consider the following aspects:

• Cost: in most cases shared instances should be more cost effective than dedicated ones.
• Legal/Operational aspects might enforce customers to run dedicated instances. Still Cloud Logging provides many security-level features (e.g. index-level security) which might be sufficient to satisfy those.
• Analytics: putting observability data into a shared instances allows for combined analysis. Note: in contrast to Application Logging, data cannot be analyzed across service instances.
• Ingestion throughput: Remote ingestion (from Cloud Foundry) may come with bit lower throughput.
  
  

Outlook

Cloud Logging service is constantly innovating to bring OpenSearch features after passing them through SAP Product standards. Few of the highlighting features which are under investigation are:

• Migration to OpenSearch 2.x
• OIDC based Auth in addition to SAML
• Extra Large plan with possible tiering/snapshot
• Multiple data sources for visualization across multiple Cloud logging instances
• Index compression for efficient storage and performance
• Simplified default user managements and API access
• And many more…

Once the features pass investigation phase and committed for delivery you could find them published in the Roadmap explorer or What’s New section.

[akuller]

Hi @Hariharan-Gandhi , nice!

Can you share the timeline for api access and container metrics for Cloud Foundry applications. We really want/need this features.

The Roadmap is outdated.

Best regards Andre

[Hariharan-Gandhi]

Hi @akuller_q, 

Thanks for reaching out. I did answer also in the other thread.

The default way to read data from Cloud Logging would be to use the OpenSearch Dashboard. To extract data directly via API of OpenSearch - we realize the benefits of access to the backend. We are working on investigating an easier provisioning of read-only-user to allow such access, minimizing the scope of database mismanagement and securing credentials.

Container metrics for Cloud Foundry applications is also in the pipeline (earlier than the API access). Using the recent contribution in Cloud Foundry community, we are currently in the load testing phase. We aim for Q1 release but it depends on the test results. 

We will publish the timeline for rollout ones we conclude the investigation. 

Best Regards, Cloud Logging Team

[akuller]

Hi @Hariharan-Gandhi ,

Yes, we have built our own views and integrated them. Nevertheless, the customer wants a deeper integration.

This are very good news regarding CF. Is there a kind of beta programm or something similar?

[sreehari_vpillai]

Thank you. @Hariharan-Gandhi 
do you have an example of instance sharing between sub account using "user provider service" ? My query is more of - whats the structure of the UPS - ? below is how I use, along with credentials and other copied from the service key 

{
    "cloud-logging": [
        {
            "label": "cloud-logging",
            "plan": "standard",
            "name": "cloud-watch",
            "tags": [],
         }
    ]
}

[Hariharan-Gandhi]

Hi @sreehari_vpillai, 

If you are using in the Cloud Foundry cli, then the `-l` attribute to mark the syslog drain url should be specified. Please refer to the "Bind the Application to User Provided Service Using the Command Line Interface" section of the help page: https://help.sap.com/docs/cloud-logging/cloud-logging/ingest-via-cloud-foundry-runtime?version=Cloud... 

• CF docs: https://cli.cloudfoundry.org/en-US/v6/create-user-provided-service.html 
• Examples using credentials `-p` option https://docs.cloudfoundry.org/devguide/services/log-management.html 

[kailashj]

For SAP Log Service, private cloud edition, that helps collect, centralize, store, forward, and access log files from all systems and applications that are in use. It centralizes the logs from customer’s systems, applications, and services running in a private cloud. Up to 50 servers are supported per tenant.

What's the tenant recommendation. In words how many tenant we propose to our new customer. Is their a way to determine it. 

[Hariharan-Gandhi]

Hi @kailashj, 

I am afraid I do not get the context / question fully. We could jump into a call where we can consult you and update back later?

Best Regards, Cloud Logging Team

[JarrodBeltramo]

Hey @Hariharan-Gandhi,

How would a customer find out what the compliance domains are for the different hyperscalers/regions so we could plan for our landscape? 

[sreehari_vpillai]

@Hariharan-Gandhi May be an unrelated question - I am looking to log the user id ( from req.user.id ) associated to all requests logged - Any clue how I can achieve that ? 

[jreimann]

Hi @Hariharan-Gandhi,

Thanks for the helpful insights in your blog! We are using a subscription-based Global Account, but SAP Cloud Logging is not available to us. Could you clarify how we can ensure a smooth transition and continue using a logging service after the deprecation?

Thanks in advance!

Best regards
Johannes

[akuller]

Hi @Hariharan-Gandhi 

we are approaching the middle of q1, are there any details about the timeline for api access and container metrics for Cloud Foundry applications?

[mohit_bansal3]

Hi @Hariharan-Gandhi 

Many thanks for this wonderful blog and had an amazing session with us as well.

We are currently working on deep diving for all essential components for this and preparing our strategy for usages.

could you please help to give some more information for below points

1. Joint log with dedicated instance between sub accounts- In application log service , We used to get the logs from all the sub accounts with spaces where we had space developer role available. We have  dedicated instance for cloud logging in all sub accounts, how we can view the joint log via cloud logging in same dashboard with space developers roles.

2. Instance sharing between  sub accounts-How we can share the same between various sub accounts. We are able to do in same sub account with multiple spaces( via cf share ), we need to check the instance sharing in multiple sub accounts.

3.As mentioned,How the more granular access( log level) etc can be done ?

Regards

Mohit Bansal

[sreehari_vpillai]

@mohit_bansal3 Point 2 - create a user provider service , where you can use ingest url directly . One instance or logging can be used in multiple sub accounts seamlessly . We are doing so

[mohit_bansal3]

Hi @sreehari_vpillai , Many thanks for your prompt input and advice based your real time experience. We have followed the below sequence of steps and getting the error ""No SAP Cloud Logging credentials found" while deploying the application:

1. Created ups instance with parameter in other sub account :  cf cups myups -l https://<ingest-username>:<ingest-password>@<ingest-endpoint>/cfsyslog

2. Bind the user provided service (ups) instance with application in mta.yml. in that mta.yml, we  did not use any cls instance in  app and getting the above error. 

Regards,

Mohit Bansal

[sreehari_vpillai]

@mohit_bansal3 that sound about right what you have followed. 

Questions -

Is it a CAP application ? 

also the obvious , have you created a service key for your cloud logging instance and fetched the credentials from the key ? 

My mta.yaml looks like this ( and its a cap app ) , where cloud-watch is the user provided service i created in my sub account. Application logging service is in another sub account. 

  - name: cloud-watch 
    type: org.cloudfoundry.existing-service
    optional: true

 also, can you g to your cf space, spot the user provided service ? and click "update" to see if the drain URL is rightly set 

[mohit_bansal3]

Hi @sreehari_vpillai  , Many thanks for your prompt reply and we are able to map it with UPS.

There are one major potential issue we can't create the custom matric with UPS. This seems to be a major limitation as the whole idea for sharing the instances between sub account and custom metric is not supported. @Hariharan-Gandhi  Kindly advice any work around for this.

Regards,

Mohit Bansal

[yyertuganov]

Good evening @sreehari_vpillai @mohit_bansal3  - I have read your comments and seems like I have the same problem, where we are also sharing Cloud Logging Instance at the Central Shared Subaccount and based on the Ingestion UserID/Ingestion Pwd / Ingestion URL from its Service key I have created User Provided Service in the other Subaccount where we will deploy our CAP app. Then I have configured mta.yaml the same as in your example and however during deployment I get the same error: "No SAP Cloud Logging credentials found".

How have you resolved this?

[sreehari_vpillai]

@mohit_bansal3 would you please explain what do you mean by custom metrics ? its all about ingest URL afaik . Cloud Foundry just streams all the logs into the ingest URL , be the instance created directly in your sub account or shared from another sub account - what am I missing ? 

also can you paste the log drain URL you use, removing the credentials from it ? 

@yyertuganov can we see the mta.yaml , details of UPS ( screenshot or so ). ?

[yevgen_trukhin]

@mohit_bansal3 Regarding custom metrics this is how you can add them: https://community.sap.com/t5/technology-blogs-by-sap/adding-custom-metrics-to-a-cap-application-usin... I would also recommend following this mission: https://discovery-center.cloud.sap/missiondetail/4432/4718/

[yevgen_trukhin]

yyertuganov is your application cap nodejs and are you using https://github.com/cap-js/telemetry?