Run and Monitor Data Processing Engine Definitions

Learning Objectives

After completing this unit, you’ll be able to:

• Explain the different ways to run DPE definitions.
• Use Monitor Workflow Services and Data Manager to monitor DPE runs.

Run the Partner Liability Definition

Linda runs the DPE definition from either the builder or using a flow. Usually a definition is only run from the builder to process specific input variable values, which test and validate the writeback results. The values for the variables must be provided at the time of the run. 

1. Click , and select Setup.
2. Enter Data Processing Engine in the Quick Find box, and select Data Processing Engine.
3. Open an active definition that you want to run and click Run Definition.
4. Click Next.
5. Enter the input variable values.
6. Click Run Definition.

Mary requires the partner liability details at the end of every week. So Linda sets up the definition to run on a particular schedule. She creates a scheduled flow that will run the definition and writeback liability at the end of each week. 

1. Click , and select Setup.
2. Enter Flows in the Quick Find box, and select Flows.
3. Click New Flow.
4. Select Scheduled-Triggered Flow.
5. Click Next.
6. Select Freeform.
7. On the Manager tab, click New Resource.
8. Specify the following details.
   • Resource Type: Formula
   • API Name: RunDPE
   • Data Type: Date
   • Formula: {!$Flow.CurrentDate}-7
9. Click Done.
   
   
   
10. On the Elements tab, drag the Action element onto the canvas, and specify the following details.
    • Filter by: Data Processing Engine
    • Action: Partner Liability-DataProcessingEngine
    • Label: RunDPEPartnerLiability
    • RunDate: {!RunDPE}
    • Enable Loyalty_Program_Name
    • Loyalty_Program_Name: Cloud Kicks Inner Circle
11. Click Done.
    
    
    
12. On the Start element, click Set Schedule and specify the following details.
    • Start Date: Current Date
    • Start Time: 12:00 AM
    • Frequency: Weekly
13. Click Done.
    
    
    
14. To connect the elements, click and drag connectors (round buttons inside each element) from the Start element to the RunPartnerLiability (Action) element.
15. Click Save.
16. Enter the flow name: FlowForPartnerLiability
17. Click Save.
18. Click Activate.

Since this is a scheduled flow, Linda doesn’t manually run it. It runs based on the schedule she sets. When it runs, the DPE definition will calculate the liability for all transactions related to Ursa Major Solar for the previous week. However, to test the flow, she clicks Debug and then clicks Run. 

Monitor the Partner Liability Definition

Once the DPE has been scheduled, Linda monitors it to see the status of the definition run. When a definition runs, a batch job record is automatically created that tracks the status of the run. In the case of a failed run, Linda troubleshoots to understand the cause and fix the problem. 

1. Click , and select Setup.
2. Enter Monitor Workflow Services in the Quick Find box, and select Monitor Workflow Services.
3. Click the Partner_Liability batch job to view details of the run.
   
   
   
4. Click the Task tab to view details of the run’s task.
   
   

After the batch job record is created, the DPE syncs each of the definition's source object data. For example, Linda previously selected the Transaction Journal and Loyalty Program as two of the source objects. A separate data sync job runs for each of these objects. After the data syncs are complete, the definition runs. Linda can then view the status of each of these data syncs. 

1. Click , and select Data Manager.
2. Monitor the data sync jobs for each Salesforce object that’s selected as a data source.

After the data syncs, the DPE creates and triggers a recipe to process the records. The recipe writes the processed data to internal file stores. The DPE then reads that data and writes it back to the Salesforce objects using Bulk API V2. If there is a problem with the execution, Linda uses the recipe to understand why. If you recall, in the Data Processing Engine in Loyalty Management module, we had enabled the Data Pipelines Add On User permission set. Without this, Linda will not be able to view the recipe. The recipe is available only while the batch job is in progress. Post execution, the recipe gets deleted. Here’s how Linda views the recipe. 

1. Click , and select Data Manager.
2. Click Recipes.
3. Open the recipe of the definition that you want to investigate.
4. On the confirmation window, click Yes, Fix It.

Linda can review the records that are being processed by each node of the definition. This can help her determine whether the issue, if any, is due to the definition’s design or the org’s data.

Linda is happy to note that the definition is running successfully. Mary confirms that she can view the liability amount in the Program Partner record. 

In Conclusion

Linda is excited about the ease with which DPE executes complex transformations. She can either use code (in the form of a JSON file) or a set of nodes to create the definitions. She can also monitor and troubleshoot problems in the definition with the use of Monitor Workflow Services. Transforming data has never been easier. 

Resources

• Salesforce Help: Run a Data Processing Engine Definition
• Salesforce Help: Monitor Your Data Processing Engine Definition Runs