May Developer Challenge - AI Services - Week 4

[noravonthenen]

CHECK OUT WEEK 5 OF THIS CHALLENGE

Congratulations! You made it to week 4 of the AI Developer Challenge! This week we will continue where we left of last week and train our regression model using the dataset we uploaded in week 3 and the Data Attribute Recommendation service (DAR).

Before we start, let me give you a brief intro to machine learning so that you know what kind of model we are going to train this week. Machine Learning is a subclass of AI that focuses on the development of algorithms and models that enable computers to learn from and make predictions or decisions based on data. Traditionally, there are three main types of algorithms: supervised, unsupervised, and reinforcement learning. Furthermore, there are also different types of machine learning models like regression, classification, clustering, and association rules. These models serve various purposes and are used in different types of machine learning problems. Regression models are models that are used to predict continuous values, such as predicting a person's salary based on their years of experience and education or predicting avocado prices based on region, type, and the other values in our dataset. For the regression model we will train this week, we are using a supervised approach, as we provide labeled training data - that is a dataset that includes the avocado prices in the training dataset from past purchases to learn from for future predictions.

There are different supervised regression algorithms that you can use such as Linear Regression, Support Vector Regression, Decision Tree Regression, Gradient Boosting Regression and many more. The Data Attribute Recommendation services uses a neural network for regression that seeks to minimize the mean squared error (MSE). If you want to learn more about the regression capabilities of DAR check out this blog post.

1. By using the GET List Executables request, you can get all the Executable IDs. We will use the Executable ID for the regression model: “340abda5-d605-452a-bf4c-169249aab362”
2. Assign the Executable ID (“340abda5-d605-452a-bf4c-169249aab362”) to the trainingExecutableIdRegression variable in your environment.
3. Use the POST Create Training Configuration Regression request to create a training configuration for DAR. This way DAR knows which algorithm and which dataset and dataset schema to use during training. The datasetSchemaArtifactIdRegression and the datasetArtifactIdRegression should have been assigned from last week. If not you can use the GET List Artifacts request to get the missing values.
4. Now you need to create a training execution using the configuration from step 3. Therefore, we will use the POST Create Training Execution Regression request. If the variable trainingConfigurationIdRegression was not assigned automatically, you can assign it in the Environment yourself. It is the ID that was returned in step 3.
5. The current status of your training execution is “UNKNOWN” and the target status is “COMPLETED”. The status might change to “PENDING” and should then change to “RUNNING” as soon as the necessary resources have been allocated and the training started. You can check the status of your training by using the GET Get Execution Regression request.
6. The output of the training will be a model artifact and the training status “COMPLETED”.
7. Use the GET Get Metric Details request to check your models performance! You need to change the Variable from trainingExecutionId to trainingExecutionIdRegression as that is the ID that is being populated in step 5.
   • MSE (Mean Squared Error) is a common metric for measuring the accuracy of a machine learning model in regression problems. It calculates the average squared difference between the predicted and actual values, which tells us about the average magnitude of error in the model. A smaller MSE is more favorable as it suggests lower error magnitudes.
   • MAE (Mean Absolute Error) is another metric used for regression models. It calculates the average absolute difference between the predicted and the actual values, giving us another perspective on the error magnitude of the model. Unlike MSE, MAE isn't sensitive to outliers as it doesn’t square the errors in the calculation. A smaller MAE is again more favorable, indicating less difference between predicted and actual values.
   • MAPE (Mean Absolute Percentage Error) measures the average absolute percent difference between the predicted and actual values. It is usually used when you want to understand the error in terms of the relative size of the predictions. It is expressed as a percentage, and smaller MAPE values indicate a better fit of the model to the data.
   • A feature contribution score is a measure that assigns the importance of each variable in making a prediction and provides insights into which features are most influential in the model. If a feature has a high positive feature contribution score, it means that the feature significantly contributes to pushing the model's output higher for a given prediction. Conversely, if a feature has a high negative score, it plays a significant role in pushing the model's output lower. Knowing which features are most important can help improve your model by focusing on those variables during feature engineering.
8. SUBMIT a screenshot of your model’s metrics like this:

Stay tuned for next week to deploy your model and run inferences to see your model in action! 

Week 1 challenge

Week 2 challenge

Week 3 challenge

[Hira]

Hi @noravonthenen ,

Hope you doing well.

I followed the steps as define but I am getting below error on step 5. Would appreciate if provide guidance to resolve. 

[noravonthenen]

Hi @Hira, your model has not been trained yet. How long has the status been unknown now?

[Hira]

Hi @noravonthenen ,

Almost 30 mins now.

[noravonthenen]

@Hira, @Mikkelj I am looking into it! 

[noravonthenen]

@Hira Can you please provide the landscape and the subaccount ID?

[Hira]

Hi @noravonthenen ,

Apologies for the delay response. 

My trail sub-account id is  4bcf0fdtrail .

[Hira]

Finally, succussed 🙂.  

Here is the screenshot @noravonthenen . 

[Mikkelj]

I'm facing the same issue and has also been waiting for 1½ hour now.

[noravonthenen]

@Mikkelj @Hira So sorry that is entirely on me! I mixed up the versions of the executable ID! We want to use the latest version 3 and the ID for that is: 340abda5-d605-452a-bf4c-169249aab362. Please try to start the training again using that executable ID! 

[Hira]

Sure, let me try again.

[MioYasutake]

Hi @noravonthenen,

I am experiencing the same issue as @Hira at step 5. In the status details there's an error "Execution 'e8937ee4c1f0daee' could not be found". Can this be causing the issue?

For the environment variable "trainingExecutableIdRegression",  "340abda5-d605-452a-bf4c-169249aab362" has been set.

[noravonthenen]

Your training is COMPLETED. Can you check again with GET Get Execution Regression? And then continue with step 7?

[Jordi_C]

I'm getting stuck at exactly the same place, I've redone this weeks challenge a few times over the last couple of days and it's still not working. Any suggestions? @noravonthenen 

[Jordi_C]

The trainings status is still UNKNOWN

[Vitaliy-R]

Here are my results:

Please note I had to modify the name of the variable in the last step from `{{trainingExecutionId}}` to `{{trainingExecutionIdRegression}}` to avoid the Error 500, as `trainingExecutionIdRegression` is the name of the variable populated at the end of the step 4:

Going back to the results it was interesting to see the top 3 contributing scores (if I read the JSON properly):

37.7% Type
31.0% Region
17.4% Date (where the price - I just assume - has been changing over time)

Have a good weekend!

[noravonthenen]

Thanks for your comment Vitaliy! I tried to specify that in step 7 but thanks for making it clear! I will clarify that more in step 7 🙂

[Vitaliy-R]

Btw, because I was using VS Code with Postman extension, I did not know that the Postman application has this nice "Visualize" tab to have a nicer overview of the data from the response:

[MioYasutake]

Completed week 3. Thanks @noravonthenen and @Vitaliy-R for your assistance!

[Alpesa1990]

Hi,

I get the following error.

does it happen to anyone else?

thanks!

[noravonthenen]

Your execution is still running so the model is not finished training. Please check again 🙂

[Alpesa1990]

HI Nora!

The model has been running from yesterday. I didn´t have this error on my last check yesterday and today I have this error and still persist.

Thanks by your attention!

[noravonthenen]

Could you try to continue with step 7? I will check on your model 🙂

[Alpesa1990]

I have done again the task 3 and 4, but I get the same error. I have created a new training

But I dont´t be able to find when I want get details.

In the step 7 request, I get the next error.

Thanks in advance by your support!

[gphadnis2000]

Getting below error for step 1 please help.

[noravonthenen]

have you refreshed your token?

[gphadnis2000]

Same issue after token refresh also

[noravonthenen]

which scenario id are you using?

[gphadnis2000]

This is the scenario id 340abda5-d605-452a-bf4c-169249aab362 mentioned in step1

[noravonthenen]

That is the executable ID that you need to continue. Sorry for the misunderstanding! I already got the execution ID for you to make it easier! The correct scenario ID to see all executable IDs is this: 35cd9fbc-7290-4042-b6df-178d74c67363

[gphadnis2000]

Thanks. Let me try

[M-K]

Here's my submission for this week:

[CameronWilson]

Here is my week 4 submission

[geek]

@noravonthenen  Have managed to get myself into a knot. Last week I used the wrong Scenario ID. Having updated to the correct one and rerun the steps for week 3. Running "Get Executables" just returns, "

Not Found, Scenario executables not found. Please check the Scenario ID." Is there away to delete the previous Scenario?

[Hira]

Hi @geek ,

If you try to get list of your artifacts, there you will find the scenario id, which you can set manually in environment variables list. Hope it solve your problem.

[geek]

Finally decided to delete the whole thing and start again, eventually:

@Hira The problem might have been simply that I needed to populate the scenarioId with the scenarioIdRegression.

Stay tuned for next week to deploy your model and run inferences to see your model in action! 

---

@noravonthenen So, there will be a week 5? When will the deadline be for completing this one? Does it need to be complete by Friday or next Wednesday

[noravonthenen]

Yes! I just posted week 5! You have time until the end of next week. That's when I will check the results 🙂

[sabarna17]

@noravonthenen - Not sure about this message. Shall I check after sometime?

[Hira]

Hi @sabarna17 

I faced the same issue, noravonthenen suggest to replace trainingExecutableIdRegression value with "340abda5-d605-452a-bf4c-169249aab362" in step 2 and then do the training request.  

[sabarna17]

Got it. Thanks @Hira . There was also another prob with the below values - 

I had to execute the clean-up APIs. Also, I am curious to know about the Clean API for training jobs if there is any.

Now the Job is running and I am waiting. (fingers-crossed)