Cleanse Data and Generate Rule Specs
Learning Objectives
After completing this unit, you’ll be able to:
- Describe how the data quality cleanse skill generates cleansed data and mapplets.
- Use the data quality rule generation skill to convert natural language rule descriptions into rule specifications.
- Evaluate rule generation results including clustered rules, test data, and agent logs.
Fix Data Quality Issues
So far you’ve assessed, diagnosed, and set up rules to catch future issues. But what about data that’s already unclean today? The data quality cleanse and rule generation skills round out the Data Quality Agent. They fix existing data and turn written rule descriptions into working rule specifications.
In this unit, you follow Ana and her merchandising colleague as they use these two skills—and everything else you’ve learned about the Data Quality Agent—to work on Cloud Kicks scenarios.
Cleanse Data and Generate Mapplets
The data quality cleanse skill works on a file you upload or a dataset from your catalog. CLAIRE GPT profiles the data to assess its quality and structure, suggests cleansing rules, and gives you a preview before you approve a cleansing plan. After you approve, it identifies key data elements, detects missing or incorrect values, standardizes formats, generates cleansed data, and creates a mapplet. You can reuse the mapplet on similar datasets later in Data Integration.
Cleansing works on data you upload directly, such as a CSV or Excel file up to 25 MB. It also works on a dataset you already have cataloged in Informatica Intelligent Data Management Cloud (IDMC). Either way, the workflow is the same. Profile, plan, approve, and generate. You always review what will change before CLAIRE GPT changes it.
Ana uploads a spreadsheet of loyalty program records with inconsistent formatting. She asks CLAIRE GPT what it can do with the unclean data. CLAIRE GPT proposes a plan. Analyze the data, identify quality issues, apply cleansing rules, and preview before-and-after results. After she approves the plan, CLAIRE GPT generates cleansed data and a mapplet she can open in Data Integration.
Example prompt: “I have an unclean data set. What can you do with it?”
Generate Rule Specifications from Natural Language
The data quality rule generation skill extracts business rules from an uploaded file or prompt containing rule descriptions in natural language, sample data, or a specified catalog source. The skill then converts the rules into rule specifications in Data Quality. When generating specifications, CLAIRE GPT analyzes the critical data elements of the dataset. If you reference a dictionary that doesn’t exist yet, CLAIRE GPT creates one automatically.
This skill is especially valuable when your organization has rule documentation that was never translated into formal rule specifications. An example is a spreadsheet of business rules maintained by a data governance team. Rather than manually rebuilding each rule, you can give CLAIRE GPT that documentation directly.
Ana’s merchandising colleague has a file describing rules for validating product SKU formats, written in plain language rather than technical logic. The colleague uploads the file and asks CLAIRE GPT to generate rules from it. CLAIRE GPT proposes a plan, then validates the rules, checks for duplicates, groups logically identical descriptions into clusters, and generates a rule script and test data for each rule.
Explore how the Data Quality Agent helps you generate rule specifications from natural language.
Evaluate Rule Generation Results
After CLAIRE GPT generates rule specifications, it executes tests for each rule and provides an agent log. The log summarizes the generated rules, their distribution across data quality dimensions, and any rules that weren’t generated. Review this log before you publish because a rule that fails its test might need a revised description rather than being published as-is.
Ana’s merchandising colleague reviews the agent log for the SKU rule generation. One rule failed testing because the description was ambiguous about which SKU prefix to allow. The colleague revises the rule description and regenerates it before publishing the corrected set to Data Quality.
Wrap Up
You’ve now used the Data Quality Agent’s full range of skills, from assessing and diagnosing scores to recommending rules, cleansing unclean data, and generating rule specifications from plain language. Combined with other CLAIRE GPT agents for discovery and data integration, you’re ready to analyze data, find it, move it, and trust it.