Orchestrate Sequential Taskflows

Learning Objectives

After completing this unit, you'll be able to:

• Describe the purpose and architectural role of a taskflow in Cloud Data Integration (CDI).
  
• Combine sequential and parallel tasks inside a graphical taskflow canvas.
  
• Explain why mappings must be wrapped in a mapping task before use in a taskflow.
  
• Apply best practices for grouping and structuring tasks within a taskflow.
  
• Configure error handling for a taskflow.
  

The Orchestration Problem

Cassie, a developer in AW Computing, now has six mappings or tasks running for AW Computing:

1. mt01_LoadCustomerData: A mapping task that runs a mapping to load active customer records from a CSV → SQL Server table
   
2. m02_LoadCustomerOrders (called by mt02_LoadCustomerOrders): Mapping that joins data from a customer table and an order table and loads it into a SQL Server table
   
3. m03_RetailEnterpriseCustomers (called by mt03_RetailEnterpriseCustomers): Mapping that joins Enterprise and Retail customer data, derives the Customer Tier and Address and routes Customers to a table depending on the Tier they have been assigned.
   
4. syn_Contact_SF: Synchronization Task, loads customer data from a SQL table → Salesforce Contact object
   
5. rep_SF_Accounts_FF: Replication Task, replicates Salesforce Account data → flat file
   
6. dtf_ArchiveOrderData: Data Transfer Task, archives order and customer data from SQL Server tables → flat file
   

Each runs independently but some of them have dependencies: LoadCustomerOrders and RetailEnterpriseCustomers should only run after the Customer load completes successfully. If she runs them separately, there's no guarantee of order.

If Cassie schedules these tasks to run independently based on wall-clock times (e.g., Task 1 at 2:00 AM, Task 2 at 2:15 AM), she risks race conditions. If mt01_LoadCustomerData takes longer than expected due to network latency, mt02_LoadCustomerOrders might execute against incomplete data.

To guarantee execution order, prevent data corruption, and run non-dependent tasks in parallel, Cassie needs an orchestration engine. CDI solves this with taskflows.

What Is a Taskflow?

A taskflow lets you automate end-to-end data processes by arranging tasks into logical sequences in a graphical workflow. It supports conditional logic (branching and looping), parallel task execution, and scheduling with email notifications.

Cassie creates a simple taskflow to execute tasks in the order she defines them. If a task fails, the taskflow stops and reports the failure—preventing downstream tasks from running on bad data.

Core Rule: You cannot drop a raw mapping directly onto a taskflow canvas. Mappings must first be wrapped inside a mapping task. In a taskflow, you drag a data task step onto the canvas and assign the mapping task to it.

Creating a Taskflow

Cassie builds a unified taskflow named tsk_CustomerOrders to manage her customer pipeline.

1. Create the taskflow: Click New | Orchestration | Taskflow | Create. Name it tsk_CustomerOrders.
   
2. Add Step 1: Drag a Data task step onto the canvas and assign mt01_LoadCustomerData.
   
3. Add Parallel Branching: Drag a Parallel Paths task onto the canvas directly after Step 1.
   
   • Branch 1: Add a Data Task and assign mt02_LoadCustomerOrders.
     
   • Branch 2: Add a Data Task and assign mt03_RetailEnterpriseCustomers.
     
4. Add Step 3: Place a Data Task after the parallel paths merge and assign syn_Contact_SF.
   
5. Add Notification: Append a Notification Task step at the end of the sequence to send a completion email to Cassie.
   
6. Error Handling: On the taskflow properties tab, configure the default error behavior. For this taskflow, select Stop Taskflow on Error to prevent downstream tasks from running on bad data.
   

When executed, the taskflow runs mt01_LoadCustomerData first. After it succeeds, it triggers mt02_LoadCustomerOrders and mt03_RetailEnterpriseCustomers at the same time in parallel.

After both parallel tasks finish successfully, it runs syn_Contact_SF to write customer data to Salesforce and finally sends a completion notification email to Cassie.

If any task encounters a fatal error, the taskflow halts instantly, preventing downstream tasks from running on bad data and protecting target tables from partial or corrupted records.

Taskflow Best Practices

To keep data operations clean, scalable, and manageable, follow these enterprise design principles.

• Logical grouping: Group tasks that share strict data dependencies where the output of Task 1 feeds into Task 2, or tasks that belong to the same business domain (for example, keep customer pipelines in one taskflow and legacy archive tasks in another).
  
• Limit complexity: Avoid overstuffing a single taskflow. If a taskflow expands beyond six to eight tasks, break it into smaller sub-taskflows.
  
• Standardize naming: Use clear prefixes like tsk_ (for example, tsk_CustomerOrders or tskflw_CustomerOrders) to distinguish taskflow assets from individual tasks.
  
• Fail fast: Always configure failure steps to stop the workflow or alert administrators rather than allowing execution to proceed silently.
  

After saving and scheduling tsk_CustomerOrders to run daily, Cassie runs that taskflow to ensure it runs correctly. She can click My Jobs to review the status.

Clicking the subtasks shows the status of each task that ran.

Clicking the taskflow instance name provides a graphical map of the taskflow with a green tick to indicate the success of a task or a red x to indicate its failure for easy troubleshooting.

Cassie's pipelines now run automatically, and she can see every step. Everything looks good, until it doesn't. Cassie is practical, so she starts asking questions. What happens if a taskflow fails? How does she check its status? Where does she go to investigate problems?

Her final step is to learn how to monitor a job run and which logs to check. The next unit covers exactly that.

Resources

• Informatica Documentation: CDI Taskflows