- SAP Managed Tags:
SAP Innovative Business Solutions (SAP IBSO) developed the Coaching Insights app for the National Hockey League (NHL). Metrics are part of the app. For example, we track usage statistics. The actual tracking of the statistics is a purely technical thing. What we want to know is: What is it used for? Who is using the results? The NHL provides the app to team coaches so that they can get real-time stats during games. This means that the NHL is our customer, and they paid for the development of the app. As for who's using it and for what purpose: The NHL uses the usage statistics to see which parts of the app are most interesting to the coaches. This information enables the NHL to make business decisions about which additional features to develop and which parts of the app don't need additional investment.
Metrics play an important role in software development, not only in the cloud, but especially in the cloud. Why is that? In situations like the NHL example, the start-up of a business, or the development of any complex software system, it's crucial to establish feedback loops. This means defining a business model for a start-up or a new software feature, measuring the success, and reacting on the measured results. See also Handling Chaos and Complexity.
A metric can be any type of measurement. Examples include measuring the average shopping cart value in an online shop, evaluation of a paper-based customer survey, a unit test, or system uptime statistics. To get a feedback cycle, you'll have to define a metric first in order to measure it and then react to the results.
Eric Ries has described this Build - Measure - Learn feedback cycle for start-ups, but the same cycle can be applied to many other situations. Here, we focus on the metrics within these feedback cycles.
Let's take a look at some relevant aspects of metrics in the context of software development. To keep the feedback cycle short, the metric should be available in real time and the measurement should be automated. A paper-based survey with a three-month evaluation period doesn't meet these criteria. An automated unit test does. However, you might have good reasons to run manual tests for several days or weeks. A usability test is an example of such a case.
For a larger cloud solution, you'll have hundreds of metrics:
This list is not complete by any means. There's one aspect to keep in mind here: You'll have metrics on different levels. There is the atomic level, like that of a unit test or memory consumption of a service. However, the atomic level won't be enough. For complex solutions, you'll also need feedback cycles on a holistic level, that is, for a business process or for the entire system.
All these metrics are technical metrics. You must define them, think about how to calculate them correctly, and measure them. However, some of them do have a certain purpose in a particular context. This purpose allows you to cluster metrics. The same technical metric can be used for several different purposes:
created by:
| Editing: Ella Wittmann | Graphics: Emma Thoma | Content: Erik Scheid
Metrics play an important role in software development, not only in the cloud, but especially in the cloud. Why is that? In situations like the NHL example, the start-up of a business, or the development of any complex software system, it's crucial to establish feedback loops. This means defining a business model for a start-up or a new software feature, measuring the success, and reacting on the measured results. See also Handling Chaos and Complexity.
SAP's big investment in metrics: In 2019, SAP acquired Qualtrics for $8 billion USD. Qualtrics offers metrics for customer, employee, brand, and product experience to enable Experience Management. Experience Management (XM) is the process of monitoring every interaction people experience with a company in order to spot opportunities for improvement.
A metric can be any type of measurement. Examples include measuring the average shopping cart value in an online shop, evaluation of a paper-based customer survey, a unit test, or system uptime statistics. To get a feedback cycle, you'll have to define a metric first in order to measure it and then react to the results.
Eric Ries has described this Build - Measure - Learn feedback cycle for start-ups, but the same cycle can be applied to many other situations. Here, we focus on the metrics within these feedback cycles.
Let's take a look at some relevant aspects of metrics in the context of software development. To keep the feedback cycle short, the metric should be available in real time and the measurement should be automated. A paper-based survey with a three-month evaluation period doesn't meet these criteria. An automated unit test does. However, you might have good reasons to run manual tests for several days or weeks. A usability test is an example of such a case.
For a larger cloud solution, you'll have hundreds of metrics:
- Hundreds of unit tests
- Several architectural fitness functions
- Usability tests
- Several business metrics
- DevOps metrics to measure the development efficiency
- Many operational metrics such as uptime statistics, mean time to recovery measurement, or resource consumption tracking.
This list is not complete by any means. There's one aspect to keep in mind here: You'll have metrics on different levels. There is the atomic level, like that of a unit test or memory consumption of a service. However, the atomic level won't be enough. For complex solutions, you'll also need feedback cycles on a holistic level, that is, for a business process or for the entire system.
All these metrics are technical metrics. You must define them, think about how to calculate them correctly, and measure them. However, some of them do have a certain purpose in a particular context. This purpose allows you to cluster metrics. The same technical metric can be used for several different purposes:
- (Customer's) Business metrics: These metrics make it possible to measure the customer's business value. They help customers drive their business. Typically, as a solution provider, we enable customers to measure these metrics, but the customers perform the actual measurement and monitoring themselves.
- Pricing metrics: These metrics are relevant for the pricing of the solution. For example, the services might be priced by memory or CPU consumption. Another example would be to count the number of sales orders created if that's relevant for pricing. The system availability also has an impact on the pricing.
- Usage metrics: These are metrics that allow you to measure the usage of a solution. For example, this might be the usage of a particular feature, as in the NHL example, or business usage measuring the number of sales orders created or the number of airports using a refueling solution. These metrics can be used for renewal discussions or to ensure that "business as normal" is possible in operations. Unlike the customer's business metrics, usage metrics are monitored by SAP - with the consent of the customer.
- Fitness functions: In the context of an Evolutionary Architecture, you might want to protect certain architectural characteristics, such as a defined performance of a business process.
- Operational metrics: These are all metrics that are relevant for operating the system. This is what you want to see in the operations dashboard. Uptime, memory consumption, or CPU consumption could be used here as well. A difference might be that you would start from the overall memory consumption of the system and drill down to specific services.
- Experience metrics: This is a new field for SAP and IBSO: metrics for customer, employee, brand, and product experience. For IBSO solutions, they can also be used to enable Experience Management.
created by:
| Editing: Ella Wittmann | Graphics: Emma Thoma | Content: Erik Scheid
