Hi everyone,
I am designing a Data Cloud flow where a CRM staging object is regularly purged due to retention rules. In Data Cloud, we need to keep the processed data in a permanent target table for audit purposes, even after the source records are deleted.
With UI-based Batch Data Transforms, platform-managed Full Refreshes periodically rebuild the target table from scratch based only on current source data. This would wipe out our historical target records once the source is purged.
I want to confirm how Python Custom Code Extensions (PySpark SDK) handle this:
- When using client.write_to_dmo(..., write_mode=WriteMode.APPEND), does PySpark strictly append new rows without ever wiping or truncating the target table?
- Is PySpark execution completely immune to automatic platform Full Refreshes, meaning Data Cloud will never automatically wipe and rebuild a target table written via Custom Code?
- Under WriteMode.MERGE, if a record is deleted from the source table, can we confirm it remains untouched in the target table rather than being deleted?
Has anyone implemented this pattern for keeping permanent history from short-lived staging sources in Data Cloud?
Thanks!
#Data360 #Automation
Hi Alessandro,
Regarding your design pattern for preserving historical audit records from short-lived staging sources in Data Cloud, here are the technical confirmations for how PySpark Custom Code Extensions handle write operations:
1.WriteMode.APPEND Execution:Append Behavior.
When using client.write_to_dmo(..., write_mode=WriteMode.APPEND), PySpark strictly appends new incoming rows to the target Data Model Object (DMO) or Data Lake Object (DLO) without truncating or wiping existing data. It will never automatically rebuild or clear out historical rows.
2.Immunity to Full Refreshes:Platform Independence.
PySpark code execution is completely separate from the platform-managed Full Refresh schedules used by standard UI Batch Data Transforms. Data Cloud will not automatically trigger a full rebuild or drop table operation on targets populated via Custom Code Extensions.
3.WriteMode.MERGE and Source Deletions:Merge Behavior on Deletions.
Under WriteMode.MERGE, records that are deleted or purged from the source staging table remain untouched and preserved in the target table. Merge updates or inserts matching records based on keys, but it does not propagate source deletions to remove historical records from the target.
