Get Started with Streaming Data Transforms in Data Cloud

Learning Objectives

After completing this unit, you’ll be able to:

• Describe what a streaming data transform is and its primary use cases.
  
• Create a streaming data transform in Data Cloud.
  

In this module, we assume you’re a Data Cloud administrator with the proper permissions to create streaming data transforms.

What Is a Streaming Data Transform?

A streaming data transform lets you clean and enrich your data in near real-time, as it enters the system. To modify select amounts of data on a scheduled time interval, use a batch transform.

For example, a streaming data transform is ideal for detecting credit card fraud. By aggregating data coming in from a vast number of processing systems, as it's ingested, and normalizing it into a standard format, they can spot irregularities in near real time.

Streaming Data Transforms Under the Hood

New records are ingested and are immediately transformed and appended to the output object.

A streaming data transform reads records from a source data lake object (DLO) and runs a SQL query that modifies incoming data. It then maps the target data lake object to the Data Cloud data model in a Data Model Object (DMO).

Create a Streaming Data Transform

1. In Data Cloud, click the Data Transforms tab.
   
2. Click New.
   
3. Choose Streaming Data Transform.
   

Complete the remaining setup screens including naming your streaming data transform, choosing a target DLO, writing a SQL query, and mapping.

Example: Normalize Phone Contacts

You can use a streaming data transform for normalization. Default Data Cloud functionality doesn't store multiple phone numbers for a contact. The DLO Contact_coreis directly mapped to the DMO Contact Point Phone and only one contact point can be mapped.

In the external systems, we currently have:

But customer support reps need near real-time access to all of a contact's phone numbers. To make all three types available, we can use a streaming data transform to normalize phone contacts.

To map multiple contact points to Contact Point, use a SQL transform with the UNION operator.

SELECT CONCAT(Contact_core.CustomerId, "_Mobile") as PhoneId,
    Contact_core.CustomerId as CustomerId,
    Contact_core.MobilePhone as PhoneNumber,
    "Mobile" as PhoneType
FROM Contact_core
WHERE ISNOTNULL(Contact_core.MobilePhone) AND Contact_core.MobilePhone <> ""
UNION
SELECT CONCAT(Contact_core.CustomerId, "_Home") as PhoneId,
    Contact_core.CustomerId as CustomerId, Contact_core.HomePhone as PhoneNumber,
    "Home" as PhoneType
FROM Contact_core
WHERE ISNOTNULL(Contact_core.HomePhone) AND Contact_core.HomePhone <> ""
UNION
SELECT CONCAT(Contact_core.CustomerId, "_Work") as PhoneId,
    Contact_core.CustomerId as CustomerId,
    Contact_core.WorkPhone as PhoneNumber,
    "Work" as PhoneType
FROM Contact_core
WHERE ISNOTNULL(Contact_core.WorkPhone) AND Contact_core.WorkPhone <> ""

After execution, the target DLO Phone Numbers contains records for each phone number type. Now, you can map the target DLO Phone Numbers to the DMO Contact Point Phone. DMO Contact Point Phone contains all three phone number types for each customer. Even better - data is available for downstream tools that consume the data model object, such as calculated insights, identity resolution, and segmentation!

Monitor Status

To monitor the status of a streaming data transform, review its processing metrics.

1. On the Data Transforms tab, click the name of the streaming data transform.
   
2. Click Refresh History to view the number of records processed, failed, and removed.
   

Setting up a streaming data transform in Data Cloud is the first step in combining real-time data with a quick, responsive, and comprehensive view of your customer. After mapping DLO to DMO, the transformed data stream is ready for downstream analytics or processing.

Resources

• Salesforce Help: Normalize Data with UNION Use Case
• Salesforce Help: Set Up a Streaming Data Transform
• Salesforce Help: Check the Status of a Streaming Data Transform
• Salesforce Help: Data Mapping