Get Started with Streaming Data Transforms in Data Cloud
Learning Objectives
After completing this unit, you’ll be able to:
- Describe what a streaming data transform is and its primary use cases.
- Create a streaming data transform in Data Cloud.
What Is a Streaming Data Transform?
A streaming data transform lets you clean and enrich your data in near real-time, as it enters the system. To modify select amounts of data on a scheduled time interval, use a batch transform.
For example, a streaming data transform is ideal for detecting credit card fraud. By aggregating data coming in from a vast number of processing systems, as it's ingested, and normalizing it into a standard format, they can spot irregularities in near real time.
Streaming Data Transforms Under the Hood
New records are ingested and are immediately transformed and appended to the output object.
A streaming data transform reads records from a source data lake object (DLO) and runs a SQL query that modifies incoming data. It then maps the target data lake object to the Data Cloud data model in a Data Model Object (DMO).
Create a Streaming Data Transform
- In Data Cloud, click the Data Transforms tab.
- Click New.
- Choose Streaming Data Transform.
Complete the remaining setup screens including naming your streaming data transform, choosing a target DLO, writing a SQL query, and mapping.
Example: Normalize Phone Contacts
You can use a streaming data transform for normalization. Default Data Cloud functionality doesn't store multiple phone numbers for a contact. The DLO Contact_core
is directly mapped to the DMO Contact Point Phone
and only one contact point can be mapped.
In the external systems, we currently have:
CUSTOMERID
|
NAME
|
MOBILEPHONE
|
HOMEPHONE
|
WORKPHONE
|
---|---|---|---|---|
C1 |
John |
408-512-2234 |
650-342-4612 |
|
C2 |
Peter |
310-453-3421 |
915-870-9900 |
But customer support reps need near real-time access to all of a contact's phone numbers. To make all three types available, we can use a streaming data transform to normalize phone contacts.
To map multiple contact points to Contact Point
, use a SQL transform with the UNION
operator.
After execution, the target DLO Phone Numbers
contains records for each phone number type. Now, you can map the target DLO Phone Numbers to the DMO Contact Point Phone
. DMO Contact Point Phone
contains all three phone number types for each customer. Even better - data is available for downstream tools that consume the data model object, such as calculated insights, identity resolution, and segmentation!
PHONEID
|
CUSTOMERID
|
PHONENUMBER
|
PHONETYPE
|
---|---|---|---|
C1_Mobile |
C1 |
408-512-2234 |
Mobile |
C1_Work |
C1 |
650-342-4612 |
Work |
C2_Home |
C2 |
310-453-3421 |
Home |
C2_Work |
C2 |
915-870-9900 |
Work |
Monitor Status
To monitor the status of a streaming data transform, review its processing metrics.
- On the Data Transforms tab, click the name of the streaming data transform.
- Click Refresh History to view the number of records processed, failed, and removed.
Setting up a streaming data transform in Data Cloud is the first step in combining real-time data with a quick, responsive, and comprehensive view of your customer. After mapping DLO to DMO, the transformed data stream is ready for downstream analytics or processing.
Resources
- Salesforce Help: Normalize Data with UNION Use Case
- Salesforce Help: Set Up a Streaming Data Transform
- Salesforce Help: Check the Status of a Streaming Data Transform
- Salesforce Help: Data Mapping