Create and Evaluate an AI Model
Learning Objectives
After completing this unit, you’ll be able to:
- Describe how to create a model using Model Builder.
- Describe why you use metrics to understand model quality.
- List some metrics used to assess model performance.
Create and Evaluate a Model
You’ve learned that AI models are the how behind the what. With AI models you can augment your business intelligence, speed up your workflows, enable your users to make better decisions, and more. In other words, AI models allow you to get value out of your data by telling a machine to analyze and become an expert on it, and then apply that expertise to new data.
Data Preparation
The majority of work required to build a predictive model actually lies in preparing the data. This is why data specialists or analysts should feel empowered to build a model, because they truly know the data. In its first release, the Einstein Studio Model Builder can train a model based on a single Data Model Object (or DMO) in Data Cloud.
While we aim to make this easier in the future, there are many tools already in Data Cloud that allow users to prepare and model data to form a table of data which can be represented as a single, denormalized, DMO. Here’s an example of how Batch Data Transforms can create a DMO for training a model.
Steps to Create a New Model
Create a new AI predictive model using Model Builder in seven steps with clicks, not code. To learn more, go to Steps to Create a Model in Salesforce Help.
Let’s consider this example of building a model from scratch to predict the likelihood of customer churn. Click through the steps using the Next and Previous buttons.
Variables and Observations
To review what you learned earlier in this module, a predictive model uses machine learning to predict future outcomes. When you create a model using Model Builder, Einstein not only analyzes the data and builds a predictive model, but also produces the training metrics so you can evaluate the model.
Variables
Models organize data by variables. A variable is a category of data, like a field in Salesforce or a column in a spreadsheet. Inputs known as predictor or explanatory variables are used to generate predictions.
Observations
Predictions occur at the observation level. An observation is a structured set of data, like a record in Salesforce or a row in a spreadsheet.
For each observation, the model uses a set of predictor variables as input (1) and returns a corresponding prediction (2) as output. The model can also return the top variables that have the most impact on the prediction. In this figure, the actual outcome, churn, is not yet known.
Training Metrics
The models you create from scratch in Model Builder (known as Einstein-created models) come with training metrics. Training metrics help you understand how your AI model was trained and assess its quality.
Training metrics are calculated based on the data used to train your model. For every observation that has a known (observed or actual) outcome, a prediction is calculated and then compared with the actual outcome to determine its accuracy.
Important: There are lots of different training metrics—in fact, way too many to cover in this module. Don’t worry, you don’t need to know all—or even most—of them. Training metrics include info bubbles
to help you interpret these metrics without needing to understand all the nuances and mathematics involved in calculating them. By providing a comprehensive set of training metrics, you can evaluate your model based on your needs. That way, you can assess model quality using the metrics that make the most sense for your solution.
- Accuracy tells you how good your predictions are on a scale of random guessing to overfit data leakage. You want it to be performant.
- Top predictors are the input, or predictor variables, that have the greatest impact on predicting the outcome.
- Variable distribution shows you a histogram of the actual, observed values in the data.
What’s a Good Model?
Naturally, if you’re going to be basing business decisions on the predictions that your model produces, then you want a model that’s going to be really good at predicting outcomes. At a minimum, you want a model that does a better job at predicting outcomes than what you have in the absence of a model, which is simply random guessing that results in data-deprived decision making!
So, what makes a model good? Broadly speaking, a good model meets your solution requirements by producing predictions that are sufficiently accurate to support your outcome improvement goals. Simply put, you want to know how closely a model’s predicted outcomes match up with actual outcomes.
To help you determine how well your model performs, Einstein-created models include training metrics that visualize common measures of model performance. (Data scientists recognize these as fit statistics, which quantify how well your model’s predictions fit the real-world data.) Keep in mind that models are abstract approximations of the real world, so all models are inevitably inaccurate to some degree. In fact, a “perfect” model should raise your suspicions, not your hopes.
When thinking about models, it’s helpful to consider the frequently cited statement attributed to statistician George Box: “All models are wrong, but some are useful.”
Once you’re comfortable with the quality of your model, you can activate it so it’s ready to use. To learn more go to Activate Your Model in Salesforce Help.