Apply Machine Learning Use Cases, Best Practices, and Security

Learning Objectives

After completing this unit, you’ll be able to:

• Identify common ML use cases for Salesforce and match them to appropriate SageMaker capabilities.
  
• Apply ML lifecycle best practices to ensure successful project outcomes.
  
• Explain security and governance fundamentals for ML workloads.
  
• Assess your readiness for production ML deployments.
  

Common ML Use Cases for Salesforce

Let’s explore two simple use cases that demonstrate how machine learning (ML) can add value to your CRM data.

Use Case 1: Lead Scoring

The business problem: Your sales team receives hundreds of leads every week, but not all leads are equally likely to convert. Sales reps need to know which leads to prioritize so they can focus their time on the most promising opportunities.

The ML approach: Lead scoring is a binary classification problem. You’re predicting whether a lead will convert or not. The model learns from historical lead data by comparing characteristics of leads that converted versus those that didn’t. It then assigns a score to new leads based on these patterns.

Building the solution: Let’s now review three simple steps of how the ML model can be built. Note, the following steps are simplified and don’t contain the full spectrum of applicable ML features from SageMaker AI. As you improve your understanding of different ML offerings in SageMaker AI, experiment with the new tools and apply the ones that fit best your needs.

Step 1: Build your first model with Canvas

Maria, a Salesforce admin, uses Canvas Predictive Analysis. She connects directly to Salesforce Data 360, selects Lead Converted as the target column, and Canvas automatically builds the model. Within hours, she has a model at 71% balanced accuracy deployed to an endpoint.

This no-code approach lets Maria prove the value of ML quickly. Sales reps can now see lead scores in Salesforce and prioritize their outreach accordingly.

• Tools used: Canvas (with built-in Data Wrangler) and Salesforce Data 360
  

Step 2: Automate monthly updates

After 3 months, Maria notices the model's balanced accuracy has dropped from 71% to 63%. The marketing team launched new campaigns that attract different types of leads, and the model hasn’t learned about these new patterns yet. Maria is manually retraining the model each month, which takes about 2 hours of her time.

To solve this, the data science team creates a SageMaker Pipeline that runs automatically on the first of each month. The pipeline extracts new lead data, trains an updated model, and deploys it only if balanced accuracy stays above 68%. Now the model stays current without any manual work.

• Tools added: SageMaker Pipelines for automated retraining
  

Step 3: Monitor for data changes

Six months later, the sales team reports that lead scores seem less accurate. The team investigates and discovers that the company expanded into a new industry, and these leads have very different characteristics. The model wasn't trained on this type of data.

They add a drift check to the pipeline. The endpoint already captures its requests to Amazon S3, so on a schedule the check compares that recent lead data against the training baseline, logs the drift scores to MLflow, and raises a CloudWatch alarm when a feature moves past its threshold. Now when significant changes occur (like entering a new market or launching a new product) the system alerts the team automatically. They can retrain the model with the new data before accuracy drops.

• Tools added: A scheduled drift check with MLflow and CloudWatch alarms
  

Use Case 2: Churn Prediction with Human Oversight

This example demonstrates how to combine ML predictions with human judgment for high-stakes decisions.

The business problem: You want to identify at-risk customers who are about to stop using your product. Management considers retention offers, but they are expensive, so you need to be confident before offering discounts or incentives.

The ML approach: Churn prediction is also a binary classification problem. In this example, you can use AWS Glue to prepare and transform large-scale data from multiple sources before training. The model learns patterns from customers who churned in the past by analyzing their usage patterns, support interactions, and account characteristics. It then applies these patterns to identify current customers at risk of churning.

Building the solution: Let’s walk through a sample scenario of how a data team builds and refines a churn prediction pipeline using SageMaker tools.

Step 1: Build the initial model

Sarah, a data scientist, uses JupyterLab with the XGBoost built-in algorithm. She prepares data using AWS Glue to extract and transform customer data from multiple sources including Salesforce, support tickets, and usage logs, while pulling reusable features like "days_since_last_activity" directly from Feature Store. After training, she achieves a balanced accuracy of 79%. She deploys to a real-time endpoint that scores customers as their data changes. The model predicts churn risk, and account managers receive alerts for at-risk customers.

• Tools used: JupyterLab, AWS Glue, Feature Store, XGBoost, real-time endpoint
  

Step 2: Add human review for uncertain predictions

After 2 months, the finance team reviews the results. They discover that 40% of customers who received retention offers weren't actually going to churn.

The team realizes the model isn't always confident in its predictions. Some customers score right at the 50% threshold so the model isn't sure if they'll churn or not. For these uncertain cases, human judgment would be more accurate.

So they build the review step themselves. A Step Functions workflow checks each prediction's churn probability before anything happens. Predictions between 40% and 70% (the cases where the model is genuinely unsure) open an approval request in Salesforce for the account manager who owns the relationship. Anything above 70% proceeds automatically. About 15 of every 100 flagged customers go to a manager, instead of all 100 getting an offer automatically. Out of every 100 flagged customers, about 27 now get an offer they didn't need, down from 40. A few customers who really would have churned slip through, but the retention budget now goes to the customers who actually need it.

• Tools added: Step Functions and a Salesforce approval process for human-in-the-loop review
  

Step 3: Switch to monthly batch scoring

The team realizes they only need updated churn predictions once per month for their quarterly business reviews. Scoring customers in real-time is unnecessary and expensive.

They switch to Batch Transform, which processes all customers at once on a schedule. They export customer data to S3 on the first of each month, run batch predictions, and import results back to Salesforce. The endpoint only runs for a few hours per month instead of continuously, reducing inference costs by more than 90%.

• Tools added: Batch Transform for scheduled batch scoring
  

ML Lifecycle Best Practices

The examples above show high-level implementation steps for specific use cases. Now let’s dive deeper into best practices for each phase of the ML project. These recommendations are based on the AWS Well-Architected Machine Learning Lens.

Data Preparation Best Practices

• Assess data quality first: Before building any model, consider generating the data quality report to identify issues and ensure you have sufficient, representative data. You can do this with Data Wrangler in Canvas.
  
• Handle missing data appropriately: Missing data is common in real-world datasets, so you need to choose the right strategy for handling it. You can use algorithms like XGBoost that handle missing values naturally, apply built-in transformations in Data Wrangler, or implement custom data cleansing logic. The best approach depends on how much data you have and why the data is missing.
  
• Engineer features thoughtfully: Create features that capture domain knowledge and business logic relevant to your problem.
  
• Version your data: Track which data was used to train each model, when data was collected, what transformations were applied, and how to reproduce the dataset. You can store versioned datasets in S3 with clear naming conventions and metadata.
  
• Handle imbalanced data: Many business problems have imbalanced classes. Ignoring class imbalance leads to models that appear accurate but fail on the minority class. For example, 95% of leads might not convert. This imbalance can cause problems if you don’t address it. Use appropriate evaluation metrics (precision, recall, F1-score, or balanced accuracy rather than accuracy alone) and consider techniques like oversampling, undersampling, or SMOTE.
  

Model Development Best Practices

• Start simple: Begin with simple models before trying complex ones. Canvas gives you an AutoML baseline without writing any code. Then add complexity only if simple models don't meet requirements.
  
• Choose the right algorithm: Match the algorithm to your problem type. Consider your data size, feature types, and interpretability needs when making your choice.
  
• Split data properly: Consider splitting your data into three sets. For example, you can use 70% for training to teach the model, 15% for validation to tune hyperparameters, and 15% for testing to evaluate final performance. This prevents overfitting and gives honest performance estimates that reflect how your model will perform on new data.
  
• Avoid overfitting: Overfitting means your model memorizes training data but doesn’t generalize to new data. You can prevent it several ways. Use regularization techniques. Keep models simple when possible. Monitor training versus validation performance. Stop training when validation performance plateaus. Use more training data if available. A model that performs well on training data but poorly on validation data is overfitting.
  
• Track experiments: Use MLflow (integrated with SageMaker Studio) to track your work. Record which hyperparameters you tried, what performance each configuration achieved, which data version was used, and how long training took. MLflow makes it easy to compare approaches, reproduce results, and understand what works best for your problem.
  

Deployment Best Practices

• Choose the right deployment option: Match your deployment method to your use case. Common options include:
  
  • Real-time endpoints for immediate predictions (like lead scoring as leads arrive)
    
  • Batch transform for periodic scoring (like monthly churn predictions)
    
  • Serverless inference for variable or unpredictable traffic
    
  • Asynchronous inference for large payloads or long processing times
    
• Test before production: Test deployments thoroughly in a nonproduction environment before going live. Verify endpoint responses, test with realistic data volumes, check latency requirements, validate prediction quality, and test error handling to catch issues before they affect users.
  
• Monitor deployment health: Use CloudWatch for metrics and alarms to catch issues quickly and maintain service quality.
  

Monitoring and Maintenance Best Practices

• Set up drift detection: Models degrade over time as data patterns change. Establish a baseline from your training data and compare incoming prediction data against it on a schedule. Log the drift scores to MLflow next to your training metrics so you can see the trend, and alert through CloudWatch when drift exceeds your defined thresholds. AWS also publishes open-source monitoring solutions you can adapt to your own datasets and thresholds rather than building this from scratch. For more information, see Amazon SageMaker Model Monitor availability change.
  
• Automate retraining: With SageMaker Pipelines, you can automate your retraining workflow. The pipeline can handle the entire process from extracting and preparing data, training a new model, and evaluating the model’s performance to deploying the model automatically if it meets your quality criteria.
  
• Version models properly: SageMaker Model Registry can help you track all model versions so you can roll back to previous versions if needed. Good versioning is essential for troubleshooting and compliance.
  

Security and Governance Fundamentals

Production ML workloads require strong security and governance. The following table summarizes key security controls, what they protect, how to implement them, and best practices. You can use this table as a reference when designing your ML security architecture.

Wrap Up

In this unit, you explored how to apply ML to real business problems. You covered several common use cases for Salesforce and saw how to progressively add SageMaker tools as requirements grow. Next, you discovered best practices for each phase of the ML lifecycle. These practices can help you build models that work well and last. Finally, you explored security and governance fundamentals.

You now have a complete foundation in SageMaker AI. You understand what ML is, how SageMaker supports it, which tools to use, and how to apply them securely. You’re ready to take the next badge and learn how to integrate SageMaker with Salesforce to bring these capabilities into your CRM.

Resources

• AWS: Machine Learning Lens–AWS Well-Architected Framework
• AWS Developer Guide: SageMaker Security
• AWS Developer Guide: SageMaker Governance