Manage Data at Scale with Salesforce

Learning Objectives

After completing this unit, you’ll be able to:

• Explain the fundamentals of data management at scale.
  
• Learn how to import your data to the Salesforce Platform.
  

What Is Data Management at Scale?

When you manage a large number of customers, you likely have large data volumes (LDVs) with millions of records. The records and data that you use include details about individual customers, transaction-related data, and data from third-party sources. As your org grows, so does the amount of data that you have to manage.

All of your data must be in the right format, must be secure, and must be readily available for users. How you manage your data impacts data accuracy and availability, how fast you can gain insights from your data, and the experience that your users walk away with. Proper data management is critical for your org’s long-term success and the value that you create for your customers.

To realize the full value of the Salesforce Platform, you must define and implement policies and processes to make sure you’re collecting the right data and managing it effectively, to keep it secure, reliable, and relevant. This is where data governance and stewardship come into the picture. 

Data Governance

Data governance establishes rules and policies to ensure reliable and relevant customer data. These rules define processes and protocols to ensure the usability, quality, and policy compliance of your data, which include:

• Creating, accessing, and updating data across many different sources
  
• Storing data across clouds and on-premises
  
• Providing data access to apps and monitoring tools
  
• Ensuring data privacy and security using data portability and “right to be forgotten” concepts
  
• Archiving and destroying data in accordance with retention schedules and compliance requirements and regulatory requirements
  

Data Stewardship

Data stewardship puts tactical roles and activities into place to support and ensure adherence to the data governance plan. The most important roles are data steward and data custodian. A data steward is responsible for data quality and works with data administration processes, policies, and guidelines. A data custodian is an IT partner who understands how the data is stored, structured, and accessed. In many cases, responsibilities are shared between the data steward and data custodian. Make sure you fill these roles with people who are familiar with the business and how your data will be used.

Create Your Data Management Strategy

To get started with data management, your Salesforce organization should have a data management strategy. Making a data management strategy doesn’t have to be complex—it depends on your organization’s needs. You may write a basic policy document that defines how you handle data or create a data governance group that will meet regularly with a well-defined submission, review, and approval process.

Whatever strategy you choose, make sure you create data management policies that help you better understand your data and ensure you comply with relevant data regulations and laws. At a minimum, your data management strategy should answer: 

• What data do you need to capture and store?
  
• Where will you store the data?
  
• How much data will you store? What is the potential growth rate of your data?
  
• Is data moving between, in, and out of systems?
  
• How will you manage and resolve record duplication?
  
• How will you manage data ownership in the platform and from a business perspective?
  
• Who needs access to what data? What is their security model?
  
• Does the data need to be masked or encrypted at rest or in transit?
  
• What is your delete/archive strategy (that is, when will redundant data be removed, and how will we delete or archive data that is no longer needed)?
  

Once you have a data management strategy in place, the next step is to get your data into the Salesforce Platform. 

Import Your Data Into Salesforce

With a data management strategy in place, you’ll understand the type and amount of data that you have, who will be working with it, and have a secure repository to store, govern, discover, and share your data. Now let’s look at how to get your data on the Salesforce Platform in the most efficient way possible.

Salesforce provides best practices and tools for the data import process. This includes tools to ingest the data, catalog, and index it for analysis, secure and protect it, and connect it with analytics and machine learning tools.

Step 1: Prepare Your Data

Because your data comes from multiple sources, much of it will be in different formats. You need to determine what data needs to be in Salesforce, which is especially important for large text fields. Here are a few tasks that you’ll need to perform before you actually load your data into Salesforce.

1. Determine the data that is a good fit. Answering “Yes” to one or more of the following questions implies that this data may be CRM-relevant, and requires more careful consideration.
   

1. Will Salesforce be the information storage and retrieval system also known as “the system of record” for this data?
   
2. Will this data be required for key business processes that are implemented on Salesforce?
   
3. Are there automation use cases (within Salesforce) that are based on changes to this data?
   
4. Are there any Salesforce-specific reports, dashboards, and KPIs (key performance indicators) that use this data?
   
5. Will users need to have activities, tasks, Chatter feeds, or Slack threads around this data?
   

2. Map existing data fields to fields in Salesforce.
   
3. Generate CSV-formatted data export files using your existing software that contain all the information that you want to use in Salesforce.
   
4. Determine the order of data loading. Parent records should be loaded first, followed by child records. Use the parent record ID for data set ordering to avoid locking.
   
5. Clean up your data to minimize errors
   

1. Try to clean and de-duplicate as much of your data as possible prior to loading.
   
2. After loading, use Salesforce tools to help you manage duplicate records and integrate third-party data. Check out the Data Quality module on Trailhead for more information.
   

Step 2: Evaluate Loading Options

Salesforce provides multiple options for loading data, based on the Salesforce edition you are using and the number of records you need to import. Here are some data import options. 

For a large data volume (LDV), Salesforce recommends using the Data Loader or an extract, transform, and load (ETL) tool configured to use the Bulk API. Consider setup options that defer non-critical processes and speed LDV loading. Minimizing barriers to data loading increases the speed of the import—just make sure that the essential pieces of your data load configuration are left intact. 

Step 3: Perform Sandbox Testing 

To make sure that your data loading process goes as smoothly as possible, Salesforce recommends using a full copy sandbox environment for testing your data and load configuration.

Full copy sandboxes are a replica of your production org, including all data, such as object records and attachments, and metadata. Full copy sandbox testing can uncover errors and save a great deal of time and effort when you perform the actual data load. Errors commonly uncovered during sandbox testing include locking issues, duplicate records, and errors in your data.

Step 4: Load Your Data 

After you’ve sandbox tested a portion of your data and resolved any errors, you are ready to load your data into the Salesforce Platform.

Regardless of the tool you select to load your data, you will:

1. Choose the destination object for your import.
   
2. Select your matching criteria to prevent duplicates.
   
3. Specify your source file.
   
4. Map your fields.
   
5. View error logs to check for problems.
   The data import wizard and the data loader both produce a CSV file with the results of your import, so you can see what went wrong with any records that failed to load.
   Depending on the version of Salesforce that you are using, you can view the status of your job, and download a copy of the results.
   
6. Check your live data when the import completes.
   

Step 5: Schedule Ongoing Data Loads

After your initial data load, start thinking about how new data will be imported. You need to identify new data that was created or modified since the last time the load process ran and import this data on a specified schedule. Some data requires real-time or near real-time syncing, while other data can be imported using a bulk update of records or batch jobs that run nightly or at scheduled intervals. For large volumes of data that result in a larger than usual volume of updates, you can follow periodic loading monthly, quarterly, or annual. 

With the Data Loader batch mode or an ETL tool, you can schedule regular data synchronization and automate the synchronization process.

In addition to the frequency of ongoing data loads, consider potential data growth and take that into consideration when planning your ongoing data loading strategy. For example, if your initial data load contained 10,000 records, and you’re planning to build an integration to your ERP system that will add an additional 2,500 records per month, the total number of records in your org is going to double in 4 months. Proper planning ensures that your tools and infrastructure keep pace with your data growth.

You now know the importance of a data management strategy and understand the high-level steps involved in getting your company’s data into the Salesforce Platform. Next, you delve into what it takes to ensure that your data management strategy remains effective and your org delivers the performance that your users expect.

Resources

• Trailhead: Lightning Platform API Basics
  
• Trailhead: Data Management
  
• Developer Docs: Best Practices for Deployments with Large Data Volumes
  
• Salesforce Help: Import Data Into Salesforce
  
• Salesforce Help: Running in Batch Mode