UI Automation

Analogy categoryAnalogies

How UI Automation Targets On-Screen Controls

UI automation is the mechanism underneath every desktop flow action that clicks, types into, or reads from a control: rather than clicking raw screen coordinates, Power Automate Desktop builds a hierarchical 'UI element tree' for a target application, similar to a DOM tree for a webpage, and each action stores a 'selector' describing the path from the root window down to the specific control. This means the same 'Click' action keeps working even if the window is resized, moved, or repositioned on screen, because the selector resolves by the control's place in the hierarchy and its properties (name, automation ID, class) rather than by fixed x/y coordinates. The UI Explorer tool in PAD lets you hover over any control in a running application to inspect its full selector path and individual properties before an action is even recorded, which is essential for building automation against applications the recorder alone struggles to capture cleanly.

🏏

Cricket analogy: Like a fielding position defined relative to the batter's stance (short leg, gully) rather than a fixed GPS coordinate on the ground, a selector locates a control relative to its place in the UI tree, not a fixed screen pixel.

Selector Editor, Anchors, and Dynamic Attributes

Raw recorded selectors often embed brittle attributes like a numeric index ('Instance = 3') that break the moment a list reorders or a new row is inserted; the selector editor lets you replace index-based matching with a stable attribute such as 'Name' or 'AutomationId' when the target application exposes one, and you can mark individual attributes as 'variable' so a single selector can dynamically match different rows or tabs at runtime by injecting a Power Automate variable into that attribute. For grid-like UIs where no attribute uniquely identifies a row, an 'anchor' element, a nearby stable control used as a relative reference point, lets you say 'the button next to the row containing this exact order number' instead of relying on a fragile absolute index. PAD's fuzzy selector matching can also tolerate minor attribute drift (for example, matching on partial text) when exact matching is too strict for an application whose labels change slightly between versions.

🏏

Cricket analogy: Like replacing 'the third fielder from the boundary' with 'point' as a fixed fielding name, swapping an index-based selector for a named attribute like AutomationId makes the target resolve correctly regardless of reordering.

Fallback Targeting: Coordinates and Relative Positioning

When an application renders through a technology PAD can't fully inspect, such as certain Java Swing apps, some game engines, or heavily virtualized Citrix sessions, selector-based targeting may fail entirely, and PAD falls back to coordinate-based actions like 'Mouse click' at a fixed screen position or relative-to-window position. Relative positioning (offset from a window's top-left corner rather than the absolute screen) is more robust than absolute screen coordinates because it survives the window being moved, though it still breaks if the window is resized or the target's layout shifts internally. For the most demanding cases, PAD's 'Get details of window' plus custom offset math, or scripting actions calling into UI Automation APIs directly via a .NET/PowerShell script, provide an escape hatch when no built-in action reliably targets the control.

🏏

Cricket analogy: Like marking a fielding drill spot with cones on the ground rather than describing it by role, absolute screen-coordinate clicking works only until the practice pitch (window) is moved, unlike a role-based selector.

text

// Example selector with a variable attribute for a dynamic row target
// (edited in the PAD Selector Builder)
<html app='chrome.exe' title='Order Portal' />
  <table id='OrdersGrid' />
    <tr row='%CurrentRowIndex%' />   // marked as variable, injected at runtime
      <ctrl name='ApproveButton' automationid='btnApprove' />

// Loop injecting the variable before each click
LOOP FOR EACH CurrentRow IN OrderRows
    SET CurrentRowIndex = CurrentRow.Index
    UI Automation: Click 'ApproveButton' using selector above

The UI Explorer's 'Highlight' and 'Freeze' features let you hover over overlapping or dynamically shown controls (like a dropdown menu) and inspect their exact selector before you ever run the flow, which is far faster than trial-and-error editing after a recording fails.

Coordinate-based and relative-position clicks assume a fixed screen resolution and window size at design time; running the same flow on an unattended machine with a different resolution, scaling factor, or maximized-vs-windowed state can cause clicks to land on the wrong control entirely, so lock the target window's size at the start of the flow whenever coordinate-based actions are unavoidable.

UI automation targets controls via a hierarchical UI element tree and selectors, not raw screen coordinates, so it survives window moves and resizes.
The Selector Editor lets you replace brittle, index-based attributes with stable ones like AutomationId or Name.
Marking a selector attribute as variable enables one selector to dynamically target different rows, tabs, or items at runtime.
Anchor elements provide a stable nearby reference point for targeting rows in grids with no unique identifying attribute.
Coordinate-based and relative-position clicks are fallbacks for applications PAD can't fully inspect, and are the most fragile targeting method.
The UI Explorer lets you inspect a control's exact selector and properties before recording or editing an action.
Locking the target window's size and resolution reduces the risk of coordinate-based clicks landing on the wrong control.