Hi Trailblazer Community,
We recently launched Einstein Decisions in Marketing Cloud Personalization using a Server-Side Campaign
, and I’m looking for insight from others who have experience with the initial learning/training period.
We have approximately 10 active promotions mapped to the same content zone and are optimizing toward Click
. Within the first 24 hours after launch, we noticed that one promotion was receiving a significantly larger share of impressions than the others.
We initially questioned whether this indicated a configuration or implementation issue, but after reviewing our Einstein Decisions Report, the behavior appears consistent with the model’s explore/exploit approach. Our understanding is that Einstein:
- Uses a portion of traffic for exploration, where promotions are served randomly to gather unbiased performance data.
- Uses the remaining traffic for exploitation, where the model serves the promotion it currently predicts will perform best.
- Continues collecting interaction data and periodically retrains, so promotion distribution may be heavily skewed—particularly early on—and can change as additional data is collected.
Our report also shows that the current winning model has a strong preference for one of the promotions, which appears to explain the impression distribution we’re seeing.
One additional observation: the training report included historical data associated with 15 promotions, although only 10 are currently active. Five promotions had been deactivated prior to our current launch.
Questions for the Community
For those who have implemented Einstein Decisions:
- Is heavily skewed promotion distribution expected during the initial learning/training period, even when the active promotions have the same eligibility/configuration?
- Is there a typical amount of time, number of interactions, or number of training cycles you recommend allowing before evaluating whether the model has sufficiently stabilized?
- How frequently does Einstein Decisions automatically retrain/redeploy its model, and is there anywhere in MCP where that cadence can be viewed?
- As additional interaction data is collected, should we expect the model to continue exploring alternative promotions, even if one promotion is currently being heavily favored?
- When promotions are deactivated, is their historical interaction data intentionally retained and used in subsequent model training? If so, how long can that historical data continue to influence the model?
- Are there specific reports or metrics you recommend monitoring during the initial learning period to distinguish expected model behavior from a configuration/eligibility issue?
At this point, we’re planning to allow the model to continue gathering interaction data and monitor subsequent training/model results rather than making premature configuration changes.
Would appreciate any guidance or lessons learned from others who have gone through the initial Einstein Decisions learning period.
Hi Micheala,
What you’re seeing can be consistent with how Einstein Decisions balances exploration and exploitation. A promotion that initially performs better can receive a larger share of traffic as the model gains confidence, so equal eligibility does not necessarily mean equal impression distribution.
I’d monitor the Einstein Decisions Report over multiple training cycles rather than judging the model from the first 24 hours. In particular, look at impressions, clicks/CTR, exploration vs. exploitation behavior, and how the model’s preference changes over time.
For the five deactivated promotions, I’d verify the training-data behavior in the Einstein Decisions documentation for your specific MCP configuration. Historical data can be relevant to model training, but the exact retention and influence period is something I wouldn’t assume without confirming the current product documentation.
If the distribution remains heavily skewed while the performance data does not support the preference, then I’d investigate eligibility, content-zone configuration, audience/context attributes, and campaign setup before concluding that the model is behaving incorrectly.
Your approach of letting the model collect more data while monitoring the training reports sounds reasonable. If you can share the specific Einstein Decisions Report metrics from the first few training cycles, the community may be able to help determine whether the behavior looks expected.
