Navigate Data Lags

Learning Objectives

After completing this unit, you’ll be able to:

• Describe the five stages of the data ingestion workflow in Marketing Cloud Next.
  
• Identify the expected time lag at each stage and how it affects campaign readiness.
  

Data Ingestion Workflow

It's Friday afternoon and the marketing team wants to launch the campaign Monday morning to capture the weekend purchases. They want to know if they can include Saturday and Sunday buyers in the campaign.

Linda knows the segment is published and the data graph is active. But answering this question isn't about checking configuration, it's about understanding how data moves through the system.

Each stage in the data workflow runs on its own schedule, and those stages run sequentially. A purchase made on Saturday morning doesn’t instantly become a flow-eligible, personalized recipient. It moves through several steps before it’s ready.

Here’s a high-level view of that workflow:

• Data stream ingestion to DLOs: Source data flows into Data 360 and lands in a DLO.
  
• DMO mapping: The raw DLO record is mapped to a standardized DMO, making it usable for segmentation and identity resolution.
  
• Identity resolution to Unified Individual: Records from multiple source systems are stitched into a single unified customer profile.
  
• Segment refresh: Newly resolved profiles are evaluated against segment criteria and added to the audience.
  
• Data graph refresh: Personalization fields, like loyalty tier, are updated in the data graph and become available for emails and decision splits.
  

To give the team a reliable answer, Linda needs to understand how long each stage takes.

Stage 1: Data Stream Ingestion

The workflow begins when a customer completes a purchase. That record must move from Cloud Kicks’s source systems into Data 360, where it lands in a DLO.Before looking at timing, it helps to understand how data gets into Data 360 in the first place.

There are two approaches.

• Standard ingestion physically copies data from a source system into Data 360's data lake.
  
• Zero copy doesn't copy anything. Instead, it connects directly to an external data platform and reads the data where it already lives.
  

Zero copy offers two ways to access data.

In live query mode, Data 360 fetches data on demand, but it only updates the data graph after a scheduled refresh.But it doesn’t update your data graph in real time. Data only becomes available after a scheduled batch run, which may take several hours depending on your data graph refresh schedule.

In acceleration mode, Data 360 stores a cached copy that refreshes on a set schedule, even every 15 minutes. This keeps data more up to date but involves storing it in Data 360.

Your choice affects how fresh your data is and when it appears in the data graph. If your org uses zero copy, factor that into your refresh planning. To learn more about zero copy, check out the Data 360 with Zero Copy Trailhead badge.

Cloud Kicks uses standard ingestion for all their sources, so that's what Linda is working with.

With standard ingestion, the tool used to copy data depends on where that data is coming from. For data already in Salesforce, the CRM connector is the native path. For data coming from outside Salesforce, like Cloud Kicks's loyalty app, the Ingestion API handles the job.

The CRM connector uses Change Data Capture (CDC) to detect and sync changes. When CDC is supported, records stream into Data 360 and typically land in the DLO within about 3 minutes. If CDC isn’t supported, or if certain fields such as formula fields are included, the system switches to batch automatically, checking for updates approximately every 10 minutes.

Keep in mind that the first time a data stream is set up, the initial extraction can take up to 24 hours. After that, updates are incremental.

Loyalty data from the external app arrives on a scheduled batch cycle, which may introduce a delay of several hours.

Saturday buyer status: Purchase data arrives in the DLO within minutes. Loyalty tier data may arrive hours later, depending on the batch schedule.

Stage 2: DMO Mapping

After the record lands in a DLO, it’s mapped to a DMO as part of the same ingestion cycle. No extra wait time here.

But this stage is where configuration gaps can cause delays that look like timing problems. Linda ran into this earlier when the loyalty tier field was missing from her decision split. The data was sitting in the DLO, but the problem was a missing mapping. If a field seems to be missing downstream, the data graph is the first place to check. For this campaign, Linda has already verified the mappings.

Saturday buyer status: Mapped to the Sales Order DMO within the same ingestion cycle.

Stage 3: Identity Resolution to Unified Individual

After mapping, identity resolution stitches records from different source systems into a single Unified Individual profile. This process runs on an automatic schedule, approximately once per day, or more frequently for smaller sets of changed records. For Linda's scenario, this means a Saturday purchase could be waiting a few hours or up to a day before Maya's profile is fully unified. If no data has changed since the last run, the job skips entirely.

Because timing depends on the schedule and recent change volume, identity resolution is the least predictable stage in the workflow. A record might be processed within hours, or it might wait closer to 24 hours.

Data 360 also supports real-time identity resolution for use cases that need immediate profile matching, such as personalizing a website experience the moment a known customer starts browsing.

For Linda's post-purchase campaign, the customers aren’t waiting for an immediate response. A discount offer email that arrives within a day is perfectly relevant, making standard batch identity resolution the right, more cost-effective choice.

Saturday buyer status: Resolved records into a Unified Individual within a few hours to up to 24 hours, depending on the schedule and change volume.

Stage 4: Segment Refresh

When a Unified Individual profile is available, the segment can evaluate it when it next refreshes. Linda’s segment refreshes daily. This means a customer resolved on Saturday morning may not be added to the segment until Saturday night or Sunday morning, depending on when the job runs.

When a segment finishes publishing, records are typically available within 15 to 30 minutes. Linda can also trigger a manual publish at any time. If she publishes on Sunday evening, she can capture most weekend purchases after identity resolution has completed. To learn more about segment publish schedules and Rapid Publish options, check out the Publish a Segment in Data 360 help documentation.

Saturday buyer status: Added to the segment after the next refresh following identity resolution, potentially Sunday night.

Stage 5: Data Graph Refresh

Even after a customer enters the flow, personalization fields are pulled from the data graph at send time and not from the segment itself.

Linda’s data graph refreshes daily, which works well for loyalty data that updates infrequently. She can also trigger a Refresh Now to update data immediately before sending. This is especially useful before a scheduled campaign launch. To explore all available refresh options, refer to Refresh a Data Graph.

Saturday buyer status: Loyalty tier available in the data graph within 24 hours of profile resolution, or sooner with a manual refresh.

Linda’s Answer to the Marketing Team

Linda presents a summary of her findings to the marketing team.

She concludes that Saturday buyers are very likely to be included in Monday’s campaign. Sunday buyers are less certain. A late Sunday purchase may not complete identity resolution before the Monday morning launch.

To improve coverage, Linda recommends:

• Manually publishing the segment on Sunday evening.
  
• Triggering a data graph refresh to ensure loyalty data is current.
  

Customers who miss the cutoff can be included in the next segment refresh.

Understanding each stage of the workflow helps Linda set realistic expectations and offer practical solutions.

Wrap Up

Linda started the week with two walls and no clear path forward. She ended it with a published segment, an active data graph, and a configured flow. The running shoe campaign is live. Gold and Diamond members are receiving their discount offer. Silver members are receiving free shipping. And Linda has the foundational knowledge to build every campaign that comes after it.

In this module, you explored how data flows through Marketing Cloud Next. It starts with raw source data landing in DLOs, through DMO mapping and identity resolution, all the way to segments, data graphs, and flow execution. You learned that different Marketing Cloud Next components draw from different data layers and that every stage of the ingestion workflow runs on its own clock. With that understanding, you can build campaigns confidently, troubleshoot issues quickly, and set realistic expectations with your marketing team.

Resources

• Salesforce Help: Salesforce CRM Connector
• Salesforce Help: CRM Connector Streaming
• Salesforce Help: About Identity Resolution