- SAP Managed Tags:
To know more about SAP Data Intelligence, refer here.
Metric Explorer is a UI component of Tracking Service and can be accessed from Scenario Manager.
To know more about Tracking service, refer this blog.
Metrics Explorer enables listing and viewing of run collections, runs, metrics and parameters which are captured as a part of machine learning experimentation and during training phase using Tracking SDK or Submit Metric Operator. It allows visualization and comparison of runs across run collections belonging to one or more scenario.
User can capture metrics and parameters under a run object during their experimentation or training phase either from notebook or pipeline execution. These runs can be logically grouped into run collections. The metrics explorer then lets user browse through the run collections, view the recorded metrics and parameters of the run and compare as well as visualize the data from different runs.
You can also see Metric Explorer in action in the video here.
In this blog, we will go through in detail the current set of features provided by the Metric Explorer.
As a prerequisite, user should have already captured the metrics during their experimentation or training via tracking sdk.
For our reference here I have taken the Housing Price use case, where I already created a Scenario called Demo, and experimented with two algorithms - Random Forest Regression and Lasso Regression. For each of these algorithms I have created a corresponding run collection: Rf_Reg and Reg_Lasso. All the experimentation runs for these algorithms are captured under respective run collections. We will explore and compare these experimentation runs, and plots some charts w.r.t rmse metric.
So let’s get started.
Launch SAP Data Intelligence from Launchpad and select ML Scenario Manager.
Search for the business scenario where machine learning experimentation was done and then navigate to the scenario details page. In this case scenario name is Demo
Once inside the scenario details page, click on “Open in Metric Explorer” button at the top of the page.
You will be re-directed to the Metric Explorer home page. At any point of time you can go back to the scenario details page via browser back.
In the top shell bar of the Metric Explorer application, the context of the underline scenario is always visible. In this case, we are coming from Demo scenario.
Left side of the home page displays the list of Run Collections created under this scenario. We can select one or more run collections from the list and the table on the right gets updated with all the runs corresponding to that run collection.
Run collection list also provides the metadata information associated with the run collection such as scenario name, source and source type.
As run collection is optional and user may not define them i.e. may not group runs under a collection, all those runs are grouped into default run collection. The name of such default run collection is the source name i.e. notebook or pipeline and a label “Default” is also tagged to such collection.
It is also possible to search for a specific run collection from the search box provided.
To view the details of a run, click on the arrow at the rightmost corner of the run detail table and it opens a preview of the run details. In case user wants to have a detailed look, they can select the maximize button
Run details page consists of two parts. The upper part provides all the metadata information associated with the run such as Run StartTime, Run Duration, Scenario name and version, Run Collection, Source etc. The lower part provides details of metrics and parameters captured as part of this run. User can switch between metrics and parameters by tab selection.
User can return to the run list view by closing the details page.
There can be cases where users would like to compare runs which are part of different business scenario. Such comparisons are also possible. Click on the “Edit list” button
In the pop-up window all the scenarios are listed. User can even search for a scenario where the desired run was created. Select the scenario(s) and click on “Done”. In this case we select scenario called test
Notice that the run collection list is updated and it now contains the run collections from both the scenarios Demo and test. Select the run collections and the run table gets updated with the corresponding runs.
Sorting or filtering of metric or parameter column in the run table is possible.
You can sort a column and the run table gets adjusted. In this case we have sorted rmse metric in ascending order.
Apart from viewing the run details in the tabular format, it is also possible to visualize a run or compare multiple runs. To do that select all the applicable runs you want to compare/ visualize and click on “Open Visual Board” button
Visual board page opens up and it provides two views. One is the Data view and other is Canvas. User can switch between these two views with the help of toggle button in the top right corner.
Data view provides the comparison of all the runs in the standard data format. It lists the metadata information of the run on the top, then the metrics followed by parameters associated with the runs.
In case there are multiple runs for comparison, the page provides a carousel to move across all the runs
This view is meant to compare the runs by looking at their data side by side.
Canvas helps you to visualize one or more runs. Initially, the canvas will be blank and the user can start creating charts by using “Add Chart” button on the top
There are three chart types which are supported now.
First is Metric against RunIDs. As the name suggests, this plot helps user to visualize how a Metric value varies across runs. This is helpful when user wants to compare the change in metric across the runs, in the course of their experimentation
Start by selecting the chart type option and click on “Next”.
Select the desired Metric from dropdown and click on “Create”.
In this case we would like to visualize how the rsme value changes across different runs of the experiment.
Chart gets plotted in the canvas. Click on a bar and it gives you more details in a tool-tip
Lets create another chart by selecting Add Chart option in the canvas. Second chart type is Metric against Parameter. As the name suggests this plot helps user to visualize how a metric value changes with change in parameter value. This chart is useful when user wants to know how a metric evolves with change in parameter like during hyperparameter optimization.
Start by selecting the chart type option and click on “Next”.
Select the desired Metric and Parameter from dropdown and click on “Create”. In this case we would like to visualize how the rsme value changes with the values of alpha.
Scatter chart gets plotted in the canvas. Click on a data point and it gives you more details in tool-tip
Third chart type is Metric against Timestamp. This plot helps user to see how a metric evolves over time and is useful in deep learning or complex training jobs. Example: how the accuracy changes or loss function converges.
Click on Add Chart and then select the third chart type option. Click on “Next”.
Select the desired Run and Metric from drop-down and click on “Create”. In this case, we would like to visualize how does accuracy changes in the training run.
Chart gets plotted in the canvas. Click on a data point and it gives you more details in a tool-tip
User can go about adding more plots in their canvas.
Do note that the canvas created is not persisted and is available till the user is in Visual Board page. Once the user goes back from this page, the plots are lost.
In case user wants to retain their canvas or share them across, they can export the canvas as PDF.
To do that, click on download button in the toolbar
Select the layout depending on the number of charts, select Landscape and click on Save.
Provide a location to save the file
The downloaded canvas can be viewed in a PDF viewer
To conclude:
Metric Explorer sits on top of tracking service and provides an easy, efficient and intuitive UI which enables listing and viewing of run collections, runs, metrics and parameters. It provides a rich set of features which helps to compare and visualize metrics belonging to runs of one or more run collections. And the ability to download the canvas as pdf which can be shared across.
Metric Explorer will be available in SAP Data Intelligence Cloud from DI:2003 release onwards and for On-premise release it is available from 3.0 version.
Metric Explorer is a UI component of Tracking Service and can be accessed from Scenario Manager.
To know more about Tracking service, refer this blog.
Metrics Explorer enables listing and viewing of run collections, runs, metrics and parameters which are captured as a part of machine learning experimentation and during training phase using Tracking SDK or Submit Metric Operator. It allows visualization and comparison of runs across run collections belonging to one or more scenario.
User can capture metrics and parameters under a run object during their experimentation or training phase either from notebook or pipeline execution. These runs can be logically grouped into run collections. The metrics explorer then lets user browse through the run collections, view the recorded metrics and parameters of the run and compare as well as visualize the data from different runs.
You can also see Metric Explorer in action in the video here.
In this blog, we will go through in detail the current set of features provided by the Metric Explorer.
As a prerequisite, user should have already captured the metrics during their experimentation or training via tracking sdk.
For our reference here I have taken the Housing Price use case, where I already created a Scenario called Demo, and experimented with two algorithms - Random Forest Regression and Lasso Regression. For each of these algorithms I have created a corresponding run collection: Rf_Reg and Reg_Lasso. All the experimentation runs for these algorithms are captured under respective run collections. We will explore and compare these experimentation runs, and plots some charts w.r.t rmse metric.
So let’s get started.
Launch SAP Data Intelligence from Launchpad and select ML Scenario Manager.
Search for the business scenario where machine learning experimentation was done and then navigate to the scenario details page. In this case scenario name is Demo
Once inside the scenario details page, click on “Open in Metric Explorer” button at the top of the page.
You will be re-directed to the Metric Explorer home page. At any point of time you can go back to the scenario details page via browser back.
In the top shell bar of the Metric Explorer application, the context of the underline scenario is always visible. In this case, we are coming from Demo scenario.
Left side of the home page displays the list of Run Collections created under this scenario. We can select one or more run collections from the list and the table on the right gets updated with all the runs corresponding to that run collection.
Run collection list also provides the metadata information associated with the run collection such as scenario name, source and source type.
As run collection is optional and user may not define them i.e. may not group runs under a collection, all those runs are grouped into default run collection. The name of such default run collection is the source name i.e. notebook or pipeline and a label “Default” is also tagged to such collection.
It is also possible to search for a specific run collection from the search box provided.
To view the details of a run, click on the arrow at the rightmost corner of the run detail table and it opens a preview of the run details. In case user wants to have a detailed look, they can select the maximize button
Run details page consists of two parts. The upper part provides all the metadata information associated with the run such as Run StartTime, Run Duration, Scenario name and version, Run Collection, Source etc. The lower part provides details of metrics and parameters captured as part of this run. User can switch between metrics and parameters by tab selection.
User can return to the run list view by closing the details page.
There can be cases where users would like to compare runs which are part of different business scenario. Such comparisons are also possible. Click on the “Edit list” button
In the pop-up window all the scenarios are listed. User can even search for a scenario where the desired run was created. Select the scenario(s) and click on “Done”. In this case we select scenario called test
Notice that the run collection list is updated and it now contains the run collections from both the scenarios Demo and test. Select the run collections and the run table gets updated with the corresponding runs.
Sorting or filtering of metric or parameter column in the run table is possible.
You can sort a column and the run table gets adjusted. In this case we have sorted rmse metric in ascending order.
Apart from viewing the run details in the tabular format, it is also possible to visualize a run or compare multiple runs. To do that select all the applicable runs you want to compare/ visualize and click on “Open Visual Board” button
Visual board page opens up and it provides two views. One is the Data view and other is Canvas. User can switch between these two views with the help of toggle button in the top right corner.
Data view provides the comparison of all the runs in the standard data format. It lists the metadata information of the run on the top, then the metrics followed by parameters associated with the runs.
In case there are multiple runs for comparison, the page provides a carousel to move across all the runs
This view is meant to compare the runs by looking at their data side by side.
Canvas helps you to visualize one or more runs. Initially, the canvas will be blank and the user can start creating charts by using “Add Chart” button on the top
There are three chart types which are supported now.
First is Metric against RunIDs. As the name suggests, this plot helps user to visualize how a Metric value varies across runs. This is helpful when user wants to compare the change in metric across the runs, in the course of their experimentation
Start by selecting the chart type option and click on “Next”.
Select the desired Metric from dropdown and click on “Create”.
In this case we would like to visualize how the rsme value changes across different runs of the experiment.
Chart gets plotted in the canvas. Click on a bar and it gives you more details in a tool-tip
Lets create another chart by selecting Add Chart option in the canvas. Second chart type is Metric against Parameter. As the name suggests this plot helps user to visualize how a metric value changes with change in parameter value. This chart is useful when user wants to know how a metric evolves with change in parameter like during hyperparameter optimization.
Start by selecting the chart type option and click on “Next”.
Select the desired Metric and Parameter from dropdown and click on “Create”. In this case we would like to visualize how the rsme value changes with the values of alpha.
Scatter chart gets plotted in the canvas. Click on a data point and it gives you more details in tool-tip
Third chart type is Metric against Timestamp. This plot helps user to see how a metric evolves over time and is useful in deep learning or complex training jobs. Example: how the accuracy changes or loss function converges.
Click on Add Chart and then select the third chart type option. Click on “Next”.
Select the desired Run and Metric from drop-down and click on “Create”. In this case, we would like to visualize how does accuracy changes in the training run.
Chart gets plotted in the canvas. Click on a data point and it gives you more details in a tool-tip
User can go about adding more plots in their canvas.
Do note that the canvas created is not persisted and is available till the user is in Visual Board page. Once the user goes back from this page, the plots are lost.
In case user wants to retain their canvas or share them across, they can export the canvas as PDF.
To do that, click on download button in the toolbar
Select the layout depending on the number of charts, select Landscape and click on Save.
Provide a location to save the file
The downloaded canvas can be viewed in a PDF viewer
To conclude:
Metric Explorer sits on top of tracking service and provides an easy, efficient and intuitive UI which enables listing and viewing of run collections, runs, metrics and parameters. It provides a rich set of features which helps to compare and visualize metrics belonging to runs of one or more run collections. And the ability to download the canvas as pdf which can be shared across.
Metric Explorer will be available in SAP Data Intelligence Cloud from DI:2003 release onwards and for On-premise release it is available from 3.0 version.
1 Comment
