Handle Common Production Operations Questions and Scenarios

Learning objectives

After completing this unit, you'll be able to:

• Monitor endpoints using CloudWatch metrics and Salesforce-side tracking.
  
• Optimize inference costs through instance sizing, auto-scaling, and deployment patterns.
  
• Apply security best practices for production ML workloads.
  

This unit covers the operational practices that make integrations production-ready. The following information is organized around the most common questions teams ask when running Salesforce-SageMaker integrations in production.

Monitoring

What Metrics Should I Monitor for My SageMaker Endpoints?

SageMaker endpoints report several important metrics through Amazon CloudWatch.

• Invocations: The number of requests your endpoint receives. Track this over time to understand usage patterns.
  
• ModelLatency: The time your model takes to respond, measured from SageMaker's perspective. This doesn't include network time. High latency means slow predictions, which can impact user experience.
  
• OverheadLatency: The time SageMaker adds to responses for request processing and response formatting. It's usually small compared to model latency.
  
• Invocation4XXErrors: Client errors that happen when requests are malformed or authentication fails. High 4XX errors might suggest problems with how Salesforce is calling the endpoint.
  
• Invocation5XXErrors: Server errors that happen when the endpoint fails to process requests or when the model stops responding.
  
• CPUUtilization: How much CPU your instances are using. High utilization means you might need more instances. Low utilization means you have an opportunity to optimize costs.
  
• MemoryUtilization: How much memory your instances are using. If memory is maxed out, predictions slow down or fail.
  

You'll find these in two places in CloudWatch. The invocation and latency metrics are in the AWS/SageMaker namespace, and the CPU and memory metrics are in /aws/sagemaker/Endpoints. For more information, see Amazon SageMaker AI metrics in Amazon CloudWatch.

How Do I Set Up Alarms for Critical Metrics?

CloudWatch alarms notify you when metrics cross thresholds so you can respond to issues quickly. When a metric crosses your defined threshold, CloudWatch sends notifications through Amazon SNS (Simple Notification Service). You can configure SNS to send alerts through email, SMS, or integrate with incident management systems like PagerDuty or Slack.

For more information, see Using Amazon CloudWatch alarms.

Where Can I View My Endpoint Metrics?

You can view metrics in CloudWatch dashboards where you can create custom visualizations and share them with your team. Additionally, you can view metrics directly in the SageMaker console by navigating to your endpoint and clicking the Monitoring tab.

The console displays metrics organized into three sections.

• Operational metrics (resource utilization)
  
• Invocation metrics (request patterns)
  
• Health metrics (endpoint status)
  

You can adjust the time period to look at the last hour for recent issues, the last week for trends, or the last month for capacity planning.

How Do I Monitor Prediction Usage on the Salesforce Side?

CloudWatch shows endpoint metrics, but you also need to track how predictions are used in Salesforce.

For Data 360 integration:

• Monitor prediction ingestion rates in Data 360.
  
• Track which Salesforce apps and users consume predictions.
  
• Review prediction accuracy by comparing to actual outcomes.
  
• Set up Salesforce reports to track prediction adoption.
  

For External Services integration:

• Monitor flow execution logs for External Service calls.
  
• Track API call volumes from Salesforce to API Gateway.
  
• Review Apex logs for prediction usage patterns.
  
• Set up Salesforce dashboards showing prediction metrics.
  

End-to-end latency tracking:

• Measure time from Salesforce request to prediction result (not just ModelLatency).
  
• Set SLAs based on user experience, not just technical metrics.
  
• Alert when end-to-end latency exceeds acceptable thresholds.
  

My Predictions Are Too Slow. How Can I Improve Latency?

For streaming predictions:

• Check the ModelLatency metric in CloudWatch to see if the model itself is slow.
  
• Verify that network connectivity between Salesforce and AWS isn't adding delays.
  
• Consider switching to asynchronous inference when you start pushing against the limits of a real-time endpoint, which times out at 60 seconds and caps payloads at 25 MB. Asynchronous inference handles payloads up to 1 GB and processing times up to an hour.
  
• Right-size the instance for your model, and check whether your request payload carries more fields than the model actually needs. If you're serving a large language model rather than a tabular one, SageMaker AI also offers optimization techniques like quantization and EAGLE-based speculative decoding, which predicts future tokens from the model's hidden layers to raise throughput without sacrificing quality.
  

For batch predictions:

• Verify your batch size isn't too large, which can slow processing.
  
• Check if your endpoint has sufficient instances to handle the workload.
  
• Consider using Batch Transform instead of a real-time endpoint for large offline workloads.
  

Cost Optimization

Which Inference Option Should I Choose to Optimize Costs?

Different inference patterns have different cost implications.

• Real-time endpoints cost the most because they run continuously. Use this when you have low latency workloads with predictable traffic patterns that need consistent latency and are always available.
  
• Serverless inference costs less for intermittent traffic. Use this for synchronous workloads with spiky traffic patterns where you need to automatically scale to meet workload traffic.
  
• Asynchronous inference can scale to zero. Use this for workloads that process up to 1 GB of data (text, images, video, audio) that are latency insensitive and cost sensitive. Control costs by specifying a fixed number of instances for optimal processing rate instead of provisioning for peak.
  
• Batch transform costs the least for bulk processing. Use this when you need inference for large datasets in offline processes.
  

For more information, see Inference options in Amazon SageMaker AI.

How Do I Choose the Right Instance Type for My Endpoint?

Use SageMaker Inference Recommender to find the right instance type. Inference Recommender tests your model on different instances, measures performance and cost, and recommends the optimal choice.

Begin with smaller instances and monitor their performance. If you notice performance issues or high utilization, scale up to larger instances. This approach prevents over-provisioning and helps you find the most cost-effective configuration.

For more information, see Amazon SageMaker Inference Recommender.

How Does Auto-Scaling Work and How Do I Configure It?

Auto-scaling adjusts instance count based on load. When traffic increases, it adds instances. When traffic decreases, it removes instances. This saves costs during low-traffic periods because you're not paying for idle capacity, and it maintains performance during high-traffic periods.

To use auto-scaling, configure it with target tracking policies. These policies let you set a target value for a specific metric, such as invocations per instance or CPU utilization. SageMaker then automatically adds or removes instances to maintain that target. For example, if you set a target of 70% CPU utilization and traffic increases beyond that threshold, SageMaker adds instances to distribute the load.

SageMaker AI also supports container caching, which stores model artifacts and container images on both existing and newly launched instances. This reduces scaling latency during traffic spikes because new model copies can start from cached data instead of downloading everything from scratch.

For more information, see Auto-scaling policy overview.

What Are SageMaker Savings Plans and When Should I Use Them?

AWS offers SageMaker Savings Plans where you commit to a consistent usage level. In exchange, you receive discounted rates compared to on-demand pricing. Savings Plans work well for steady workloads. If you know you'll use a certain capacity continuously, you can commit to it for a lower rate.

For more information, explore What are Savings Plans?

Should I Use Multi-Model Endpoints?

If you have multiple models, consider multi-model endpoints. Instead of one endpoint per model, you can host multiple models on one endpoint where they share the underlying instances. This reduces costs because you're paying for fewer instances, and it simplifies management because you have fewer endpoints to monitor.

Multi-model endpoints work well when models are similar in size and have similar traffic patterns. They're less effective when one model receives significantly more traffic than others.

For more information, see Multi-model endpoints.

Security

What Security Best Practices Should I Follow for Production Endpoints?

If you completed Amazon SageMaker AI and Machine Learning Essentials, you learned the foundational security practices for SageMaker integrations. Here are the key points to remember for production operations.

• IAM roles with least privilege: Grant only necessary permissions. For example, for endpoints, use sagemaker:InvokeEndpoint instead of broad sagemaker:* policies.
  
• VPC configuration: Deploy endpoints in a VPC to isolate them from the public internet and control access with security groups.
  
• Data encryption: Use encryption at rest (default in SageMaker, or your own KMS keys) and in transit (TLS/HTTPS).
  
• Secrets management: Store credentials in AWS Secrets Manager, not in code or configuration files.
  
• Access monitoring: Enable CloudTrail logging to track API calls and review logs for unusual patterns.
  

For more information, see Configure security in Amazon SageMaker AI and Best practices for endpoint security and health.

Troubleshooting

My Predictions Stopped Working and I'm Seeing 401 or 403 errors. What should I check?

For Data 360 integration:

• Confirm the OAuth connection is still valid. Access tokens refresh on their own. If the refresh token was revoked, the connected app or external client app was changed, or the stored secret was edited, refresh stops working and you need to re-authorize the connection.
  
• Verify the connected app or external client app is still active and hasn't been deactivated.
  
• Confirm your IAM role has the sagemaker:InvokeEndpoint permission.
  
• Try re-authenticating the Data Wrangler connection to Data 360.
  

For External Services integration:

• Verify API Gateway IAM authentication is configured correctly.
  
• Check that Named Credentials in Salesforce are still valid and haven't expired.
  
• Confirm your API Gateway resource policy allows traffic from Salesforce IP ranges.
  
• Test authentication independently using the AWS CLI to isolate the issue.
  

For more information, see Troubleshoot Amazon SageMaker AI model deployments and Set up OAuth connections for Salesforce Data Cloud.

Predictions Are Returning from SageMaker but Not Appearing in Salesforce. Where's the Problem?

For Data 360 integration:

• Verify your Model Builder endpoint registration is active and hasn't been disabled.
  
• Check that Data 360 field mappings are correct and match your model's output format.
  
• Confirm prediction scheduling is enabled (streaming or batch).
  
• Review Data 360 logs for ingestion errors that might indicate data format issues.
  

For External Services integration:

• Check that your flow or Apex code is correctly storing predictions to Salesforce fields.
  
• Verify the External Service action is returning data in the expected format.
  
• Review Salesforce debug logs for any errors during the External Service call.
  
• Confirm field-level security settings allow writing to your target fields.
  

How Do I Update My Model without Breaking the Integration?

For Data 360 integration:

1. Deploy your new model to a new SageMaker endpoint (don't replace the existing one yet).
   
2. Test the new endpoint thoroughly with sample data.
   
3. Update the Model Builder endpoint registration to point to the new endpoint.
   
4. Monitor for issues and keep the old endpoint running temporarily as a fallback.
   
5. Once the new endpoint is stable, delete the old endpoint.
   

For External Services integration:

1. Deploy your new model to a new endpoint.
   
2. Update API Gateway to point to the new endpoint.
   
3. Test the External Service action with the new endpoint.
   
4. If the response format changed, update your OpenAPI specification.
   
5. Re-import the External Service in Salesforce if the spec changed.
   

Blue/green deployment strategy:

1. Run both old and new endpoints simultaneously.
   
2. Route a small percentage of traffic to the new endpoint initially.
   
3. Monitor performance and accuracy closely.
   
4. Gradually shift more traffic to the new endpoint.
   
5. Keep the old endpoint as a fallback for 24-48 hours before decommissioning.
   

For more information, see Best practices for deploying models on SageMaker AI Hosting Services.

Where Can I Find Logs to Troubleshoot Issues?

Different logs serve different purposes.

• CloudWatch Logs: SageMaker endpoint errors, performance metrics, and model execution details
  
• CloudTrail logs: Complete API call history and security audit trail for compliance
  
• API Gateway Logs: Request/response details, authentication issues, and transformation errors (External Services only)
  
• Salesforce Debug Logs: Flow execution, Apex errors, and External Service call details
  
• Data 360 Logs: Prediction ingestion status and field mapping issues (Data 360 integration only)
  

For more information, see Monitoring AWS resources in Amazon SageMaker AI.

Wrap up

In this unit, you learned how to operate Salesforce-SageMaker integrations in production through practical Q&A. You explored monitoring strategies, cost optimization techniques, security best practices, and troubleshooting approaches.

Production operations require ongoing attention. We recommend that you monitor continuously from both AWS and Salesforce perspectives, optimize costs as usage patterns change, and maintain strong security practices. With these fundamentals in place, your integration will run reliably, perform well, stay within budget, and keep customer data secure.