Automate Dynamic Interactions with a Windows Program

Learning Objectives

After completing this unit, you’ll be able to:

• Confirm a page loaded correctly.
  
• Use the result of an image search as an anchor for a subsequent image search.
  

Before You Start

There are specialized skills that enable you to handle more sophisticated use cases and make your code more flexible and reusable. These skills include handling errors, looping over lists or collections, making logical decisions inside an RPA Bot Task workflow, and using more dynamic pattern recognition action steps.

We assume your organization has purchased a MuleSoft RPA license. If your organization has not purchased a MuleSoft RPA license, that’s OK. Read along to learn how MuleSoft RPA Builder users perform the steps. Don’t try to follow the steps in a Trailhead Playground, because MuleSoft RPA Builder isn’t available in the Trailhead Playground.

A programming background isn’t necessary to understand the concepts introduced in this module. However, those with a programming background may find many of the concepts familiar. If you don’t have a programming background, it may be helpful to continue this module but pair up with an RPA developer during your first RPA process implementation project.

In this unit, you learn how to set anchor points to limit the search area in an Image Search action step. Then, relative to those anchor points, you learn how to identify nearby UI elements and automate the selection of an item in a particular dropdown list.

Automate Dynamic Interactions

When humans interact with webpages, they expect changes to occur when clicking a button, typing keystrokes, and scrolling down the page. Your MuleSoft RPA bot needs to automate these dynamic interactions as well.

Use Anchor Points to Limit the Search Area

The previous module focused on searching for images in relation to the entire screen. When you do that, the search area is always in relation to the top-left corner of the screen. Even when adding a narrowed search area, the top-left corner of the search area is a constant distance from the top-left corner of the screen. However, image searches can build on each other. A second search can use the location of the first search as a reference point or anchor point. This allows the search area to be dynamic and lets the bot search relative to something else, rather than an absolute section of the screen.

The first concern is to add accuracy and performance to your search by narrowing the search area. The second, and arguably more important, concern is to handle a scenario where there are multiple identical images and to determine the correct one to select.

An example of this is when viewing a report with several rows and columns. One column has a button called Open that opens the record in the row. This button appears in every row. By using an image search to identify the correct row, you can then narrow the search area by using this location as an anchor point to identify the correct Open button.

Verify Page Loads Correctly

When viewing websites, navigation is a typical human interaction. It involves mouse actions to scroll down the page and select the hyperlink that contains the item you’re looking for. After you click the hyperlink, you may expect a new page to load.

You learned in the previous module that you can click a hyperlink by using the Mouse Actions action step. After clicking, you need to verify the page loads correctly. To verify this, you can add an additional Image Search action step to find specific text or an image that may be on the new page. If Image Search finds a match, your MuleSoft RPA bot verifies that the page loads successfully. If Image Search fails to find a match, your MuleSoft RPA bot throws an error during this task rather than the next.

Interact with Dropdowns

Dropdowns are another commonly used dynamic interaction element in websites. Interacting with dropdowns typically requires the following action steps.

1. Locate the dropdown by using an Image Search action step.
   
2. Open the dropdown by using a Mouse Actions action step.
   
3. Find an option in the dropdown by using an Image Search action step.
   
4. Set the search area in reference to the previous Image Search action step.
   
5. Click the dropdown option by using a Mouse Actions action step.
   

Walkthrough

Now that you’ve learned how to verify page loads, use anchor points, and interact with dropdowns, take a look at the following video demonstration showing how to put these skills to practice. If you wish to follow along, check out the walkthrough instructions.