Get Started with Streaming Data Transforms in Data Cloud
Learning Objectives
After completing this unit, you’ll be able to:
- Describe what a streaming data transform is and its primary use cases.
- Create a streaming data transform in Data Cloud.
What Is a Streaming Data Transform?
A streaming data transform lets you clean and enrich your data in near real-time, as it enters the system. To modify select amounts of data on a scheduled time interval, use a batch transform.
For example, a streaming data transform is ideal for detecting credit card fraud. By aggregating data coming in from a vast number of processing systems, as it's ingested, and normalizing it into a standard format, they can spot irregularities in near real time.
Streaming Data Transforms Under the Hood
New records are ingested and are immediately transformed and appended to the output object.
A streaming data transform reads records from a source data lake object (DLO) and runs a SQL query that modifies incoming data. It then maps the target data lake object to the Data Cloud data model in a Data Model Object (DMO).
Create a Streaming Data Transform
- In Data Cloud, click the Data Transforms tab.
- Click New.
- Choose Streaming Data Transform.
Complete the remaining setup screens including naming your streaming data transform, choosing a target DLO, writing a SQL query, and mapping.
Example: Normalize Phone Contacts
You can use a streaming data transform for normalization. Default Data Cloud functionality doesn't store multiple phone numbers for a contact. The DLO Contact_core
is directly mapped to the DMO Contact Point Phone
and only one contact point can be mapped.
In the external systems, we currently have:
CUSTOMERID
|
NAME
|
MOBILEPHONE
|
HOMEPHONE
|
WORKPHONE
|
---|---|---|---|---|
C1 |
John |
408-512-2234 |
650-342-4612 |
|
C2 |
Peter |
310-453-3421 |
915-870-9900 |
But customer support reps need near real-time access to all of a contact's phone numbers. To make all three types available, we can use a streaming data transform to normalize phone contacts.
To map multiple contact points to Contact Point
, use a SQL transform with the UNION
operator.
SELECT CONCAT(Contact_core.CustomerId, "_Mobile") as PhoneId, Contact_core.CustomerId as CustomerId, Contact_core.MobilePhone as PhoneNumber, "Mobile" as PhoneType FROM Contact_core WHERE ISNOTNULL(Contact_core.MobilePhone) AND Contact_core.MobilePhone <> "" UNION SELECT CONCAT(Contact_core.CustomerId, "_Home") as PhoneId, Contact_core.CustomerId as CustomerId, Contact_core.HomePhone as PhoneNumber, "Home" as PhoneType FROM Contact_core WHERE ISNOTNULL(Contact_core.HomePhone) AND Contact_core.HomePhone <> "" UNION SELECT CONCAT(Contact_core.CustomerId, "_Work") as PhoneId, Contact_core.CustomerId as CustomerId, Contact_core.WorkPhone as PhoneNumber, "Work" as PhoneType FROM Contact_core WHERE ISNOTNULL(Contact_core.WorkPhone) AND Contact_core.WorkPhone <> ""
After execution, the target DLO Phone Numbers
contains records for each phone number type. Now, you can map the target DLO Phone Numbers to the DMO Contact Point Phone
. DMO Contact Point Phone
contains all three phone number types for each customer. Even better - data is available for downstream tools that consume the data model object, such as calculated insights, identity resolution, and segmentation!
PHONEID
|
CUSTOMERID
|
PHONENUMBER
|
PHONETYPE
|
---|---|---|---|
C1_Mobile |
C1 |
408-512-2234 |
Mobile |
C1_Work |
C1 |
650-342-4612 |
Work |
C2_Home |
C2 |
310-453-3421 |
Home |
C2_Work |
C2 |
915-870-9900 |
Work |
Monitor Status
To monitor the status of a streaming data transform, review its processing metrics.
- On the Data Transforms tab, click the name of the streaming data transform.
- Click Refresh History to view the number of records processed, failed, and removed.
Setting up a streaming data transform in Data Cloud is the first step in combining real-time data with a quick, responsive, and comprehensive view of your customer. After mapping DLO to DMO, the transformed data stream is ready for downstream analytics or processing.
Resources
- Salesforce Help: Normalize Data with UNION Use Case
- Salesforce Help: Set Up a Streaming Data Transform
- Salesforce Help: Check the Status of a Streaming Data Transform
- Salesforce Help: Data Mapping