Hi Everyone,
I’m currently evaluating Einstein Recommendation Builder and have been trying to understand how the recommendation model is actually built and evaluated behind the scenes.My understanding is that Recommendation Builder takes:
- Recommended Item Object
- Interaction Object
- Interaction Type (View, Purchase, Rating, etc.)
However, I’m struggling to understand what happens after that.
- How is the model evaluated? --- for accuracy and precision
- What is Salesforce’s recommendation for validating recommendation quality before moving into production? -- is there any smart way to do it.
Recommendation Builder doesn’t appear to expose any similar metrics.
Recommendation Builder does not expose the classic precision, recall, and F1 metrics you'd get from a hand-rolled recommender or even from Prediction Builder's scorecard. That's not you missing a screen. The model evaluation in Recommendation Builder is deliberately abstracted, and the single metric Salesforce surfaces is predicted lift, not accuracy or precision. So the honest framing for your evaluation is: you validate Recommendation Builder differently than you'd validate a model where you control the train/test split.
Here's what's actually happening behind the scenes and how to work with it.
After you select your Recommended Item object, Interaction object, and Interaction types, the build runs two things you can act on. First the Data Checker, then the scorecard. The Data Checker validates whether each object has enough data to build a meaningful recommendation before it lets you proceed, and the scorecard then gives you the model's quality, top predictive factors, and predicted lift.
Predicted lift is the metric to anchor on, and it's more useful than it first appears. Lift tells you how much better your recommendations are than simply suggesting the most popular items, and the recommended workflow is to use the scorecard to measure that, and if you're not happy, fine-tune the segments and settings and rebuild until performance is acceptable. So lift is your relative quality signal: a model that can't beat a popularity baseline by a meaningful margin isn't worth deploying.
On your two specific questions.
For "how is the model evaluated for accuracy and precision," the answer is that Salesforce evaluates it internally against held-out interaction data but does not expose precision/recall to you.
For "what's the smart way to validate quality before production," since the tool won't give you offline precision, build your own holdout validation around it. This is the approach I'd take and the one worth bringing to a client:
Reserve a time-based holdout of interaction data that you exclude from the build window. Generate recommendations from the model, then check how many of those recommended items the same users actually interacted with in your held-out period. That gives you a real precision and hit-rate measurement that the tool doesn't surface, computed against ground truth you controlled. It's the closest thing to the precision/recall you're asking for, and it's defensible.
Run a champion/challenger in production rather than a hard cutover. Deploy the recommendation inside a Next Best Action strategy and A/B test it against either the existing logic or a popularity baseline, measuring actual click-through or conversion lift on live traffic. Recommendations deploy by adding business rules in an NBA strategy, which is also where you control and gate them, so the strategy layer is your natural place to split traffic and measure real-world precision through behavior rather than an offline score.
- SP
