Use Dynamic Values in Automated GUI Searches

Learning Objectives

After completing this unit, you’ll be able to:

• Add an activity parameter to a workflow.
  
• Use an Optical Character Recognition (OCR) action step to locate text on a webpage.
  

In this unit, you learn how to use an Optical Character Recognition (OCR) action step to dynamically search a GUI for a text string that matches a value set in an activity parameter. This allows the same RPA process to look for different values depending on the value set in an activity parameter.

When to Use an OCR Action Step

The Image Search action step is a powerful tool to help your MuleSoft RPA bots find reference points on the screen. However, what if you want your bot to search for one word one time and another word another time? This is a job for OCR. OCR converts the text on the page into a list of words with their respective coordinates, then checks if the phrase exists in that list.

OCR is more complex than a simple image search. It requires more customization and testing to ensure it works properly. However, OCR’s ability to search for text dynamically is worth the extra complexity. 

Implement Keyboard Shortcuts and Keystrokes

Another common human action that your MuleSoft RPA bot needs to emulate is keystrokes. Keystrokes are performed by using the Enter String action step or the Keystrokes action step.

When to Use Keystrokes Over Enter String

The Keystrokes action step allows you to record keystrokes and specify the window where the text is entered. For example, if your MuleSoft RPA bot types Ctrl+f, you can specify in which window you want the command to run. It’s recommended that you use the Enter String action step in all other cases.

Combine OCR with Activity Parameters and Keystrokes to Search for Dynamic Text

Now, you’re ready to use OCR to dynamically search lists as well as instances of a word stored in an activity parameter. Recall that you use activity parameters, or variables, to pass data into the RPA flow and between flow activities. In our use case, the webpage we are navigating contains a long list of virtual meetups and events. Within this list, we are interested in RSVPing to the first Composer event that is found. The value “Composer” is stored as an activity parameter because we would like this value to be dynamic.

First, use the Google Chrome™ web browser’s built-in search feature (Ctrl+f keystrokes) to search for the value stored in the activity parameter. This automatically scrolls the webpage to find the first instance of the value, which in this case is Composer.

However, just because the value is visible on the screen doesn’t mean that the MuleSoft RPA bot knows where the value is located. You may think that you can now use the Image Search action step to locate the value. However, it turns out that this won’t work because the value is dynamic. The Image Search action step doesn’t have the ability to search for dynamic values. 

Here’s where the OCR action step comes in handy. In the following screenshot, notice that when the Search mode is set to Fuzzy Comparison, the Search text can be set to an activity parameter. This is how the OCR action step can find dynamic values.

Now that you have identified the location of your text, you can narrow the search area by using this location as an anchor point to RSVP to the correct event.

Walkthrough

Now that you’ve learned how to search for dynamic values using OCR, take a look at the following video demonstration showing how this is done in practice. If you wish to follow along, check out these walkthrough instructions.

In this unit, you learned how to use keystrokes to interact with a webpage. You also learned how to use an activity parameter within a Bot Task workflow and how to use an OCR action step.

Chrome is a trademark of Google LLC.