Prepare for and Implement Load Testing
Learning Objectives
After completing this unit, you’ll be able to:
- Explain why load testing is important.
- List three changes that can affect site performance.
- Understand your choices when selecting a test environment.
- List the load testing best practices.
Why Load Test?
System performance and scalability can make or break a merchant’s business during the holiday season. To make sure her sites stay strong, Linda Rosenberg, Cloud Kicks admin, performs load testing early in her holiday readiness schedule. Load testing gives her an accurate picture of how her site will scale during the holiday season. Though the platform seems stable now, changes over time such as these can affect performance.
- Changes to custom code
- Changes to data configurations
- Traffic volume, both times and amount
She considered performing load testing on her production instance, but decided against it, not wanting to disrupt the shopper experience in any way. Here are some other choices.
- Maintain a dedicated realm for test and development.
- Test on a temporary rental realm.
- Hire a third-party load test vendor to help.
When she monitors her own tests, Salesforce customer support configures her realm based on agreed upon specs, and assists with errors on or related to the platform.
Rental realms are time-limited, typically up to 60 days. There’s no load testing after October 31, due to the impending holiday season. Here’s how to get a rental realm.
- Notify your Salesforce Account Team to place an order.
- Provide the dates you need the test realm (30 or 60 day rental limits).
- Provide traffic statistics so it can be sized correctly.
- Your Salesforce Account Team places an order.
- Your Salesforce Account Team notifies you once your order has been processed.
Linda can obtain a service agreement with Salesforce professional services or a partner who:
- Reviews her test plan
- Resources and configures a realm
- Monitors instance and profile custom code during load test
- Fine-tunes realm internals
- Delivers a final load test report
Phased Approach
To ensure her tests are consistent each year, Linda uses this phased approach.
- Plan: Define the scope, design, and scripts that form the foundation of the project.
- Execute: Run, adjust, and monitor the load test. Incrementally increase volume per run to uncover issues before moving to the target volume runs.
- Analyze: Integrate load test results, test environment metrics, and profiler data into a cohesive report. This phase has multiple owners.
- Resolve: Address identified areas of custom code with poor performance and engage appropriate resources to resolve problems.
Load Test Tools
Salesforce doesn’t recommend a specific load test tool for a B2C Commerce storefront. So Linda looks for an industry standard or open source tool that meets these requirements.
- Highly scalable: Uses load generation agents from a public or private cloud and executes from various geographic locations.
- Realistic load profiles: Simulates real people.
- Transaction based dynamic load adjustment: Runs tests at a specified hourly transaction rate.
- Robust scripting language: Uses a nonproprietary object-oriented programming language.
- Comprehensive reports: Provides flexible reports that easily and quickly compare runs using configurable content and layout.
Choose the Load Model
Linda’s test tool must be able to define the attributes she can influence to reach a specific load and performance behavior using one of these models.
- User count: A static and non-feedback-based load model. Load is determined by the number of concurrent users. When Linda configures a load of 10 users, for example, the tool runs 10 threads, repeatedly executing this scenario. At any time during the test, the target system must handle 10 concurrent users, no more, no less. The number of executions (transactions) that can be achieved during a certain time period depends on the average response time.
- Arrival rate: A feedback-based load model. The target number of transactions per hour determines the load generation. For an arrival rate of 1,000 requests per hour, the tool should run the scenario 1,000 times, equally distributed across a period of one hour. The tool should use as many concurrent users as necessary to fulfill the given arrival rate, but no more than specified.
Use the arrival rate load model if you want to prove that a system can handle a certain number of transactions per hour. The B2C Commerce test team uses this model to validate projected sale transaction volumes.
Concurrency
For both models, the user load upper threshold doesn’t represent actual user concurrency at any one test time period. This is due to factors such as the variability in response times to return a page request or randomized think times. Focus on transactional load (dynamic) to simulate during a time period.
Define the Test Period
The testing tool should let you filter out data that’s not representative or applicable, such as site warm up activities. Measure standard load tests for at least four hours, and less time for iterative tuning or to troubleshoot issues.
Start Small
Don’t use a full target load on the first test run. Instead, start small and increase incrementally, so that you can identify and resolve issues and performance problems that arise during the lower volume runs.
Load Test Types
Here are the basic load test types.
|
Type |
Description |
|
Baseline |
Establishes the response and characteristic baseline of the site under load. You compare configuration and code changes against this baseline. |
|
Comparison |
Compares a known state site or application to the same site or application with code changes. It lets you compare site performance after configuration and code changes against an established baseline. |
|
User Experience |
Compares the average response for a user under expected normal production traffic for a defined set of user scenarios against what happens during a spike or a short burst of high volume. |
|
Capacity |
Measures a site’s ability to support a large user load. Use it to determine the load at which storefront response is no longer acceptable. It also helps determine the volume of traffic that a customized storefront can handle with existing resources. |
|
Longevity |
Simulates sustained load over 24 to 48 consecutive hours during which transaction volume varies. This test evaluates custom storefront code to help identify things like memory leaks caused by custom storefront code over time, and the ability of third-party services to support varied load over time. |
|
Latency |
Simulates users at a low connection speed, so helps you identify areas on the storefront to improve the low bandwidth user experience. This can also help with mobile users. |
Iterative Testing
Iterative testing requires consistency and adequacy. Only meaningful test results allow you to accurately assess whether you’ve met your business requirements. For the sake of consistency, change just one load test variable per test run, such as the load model (dynamic or nondynamic), or the load profile. To ensure your tests are adequate, run several iterations.
Analyze Results
Load test results can vary from one test run to another, bringing the validity of your results into question. To increase confidence in your results, follow these standard practices.
- Allow for warm-up time: During warm-up, applications cache pages and initialize databases, after which they run in what’s called a steady state. Start measuring load at steady state.
- Understand your test sample size: Test results data usually have outliers. Make your test sample size big enough that the outliers are just white noise. Increase the sample size by testing for a longer period, or run multiple tests and average the results. Capture at least one hour of measurement data.
- Validate results consistency: Gauge test results using standard deviation. What percentage of the median does the standard deviation show? Use 80th and 90th percentile algorithms to process the data into the highest number of measurements and use those numbers.
- Investigate results trends: Review the results for the entire test run. What are the trends? Optimally, the graph should be nearly flat, reflecting a steady state. Investigate spikes and upward trends.
The tests help identify bottlenecks so that Linda can make resource or configuration changes or optimize customizations.
Best Practices
Linda is keen to follow best practices to ensure her sites achieve her performance and scalability goals. Here are some important load testing best practices.
- Notify customer support of the test.
- Develop a realistic load.
- Use the same code base, and equivalent platform resources and data as production.
- Target volume based on real data or business projections and the traffic you expect.
- Capture load test data by the number of virtual users or transactions/hour.
- Define a scenario of detailed scripted actions such as searching and browsing products or purchasing with promotions.
- Create a load definition profile that distributes a realistic target load over different actions to simulate production usage.
- Use good coding standards such as reviewing script designs before coding begins or developing modular and reusable script code.
- Use think times in scripts to mimic user activity. This is the time a user pauses to think about or look at something. Without think time, a load test iterates through actions at an unrealistically fast rate.
- Disable email sending in custom code to avoid throttling of email flood during the test.
- Include third-party service calls when possible.
Next Steps
In this unit, you learned about the importance of load testing, and how to plan for and implement it. Next, learn how to clean up your storefront data for peak performance.
