Screen Actions

Actions

Screen content, screenshots, and node interactions

Access these methods through agent.actions. Get screen content, take screenshots, and interact with UI nodes.

`screenContent()`

TypeScript

screenContent(): Promise<AndroidNode>

Gets the accessibility tree of the currently focused window. Returns an AndroidNode representing the root of the UI hierarchy.

Returns

Promise<AndroidNode>Root node of the accessibility tree

Examples

TypeScript

const screen = await agent.actions.screenContent();

// Find elements
const button = screen.findTextOne("Submit");
const allButtons = screen.filterAdvanced(f => f.isButton());
const input = screen.findAdvanced(f => f.isEditText().isEditable());

`allScreensContent()`

TypeScript

allScreensContent(): Promise<AndroidNode[]>

Gets the accessibility trees from all visible windows (useful for dialogs, overlays).

Returns

Promise<AndroidNode[]>Array of root nodes for each window

Examples

TypeScript

const screens = await agent.actions.allScreensContent();
for (const screen of screens) {
  const dialog = screen.findTextOne("OK");
  if (dialog) break;
}

`screenshot()`

TypeScript

screenshot(maxWidth: number, maxHeight: number, quality: number, cropX1?: number, cropY1?: number, cropX2?: number, cropY2?: number): Promise<{screenshot: string | null, compressedWidth: number, compressedHeight: number, originalWidth: number, originalHeight: number}>

Takes a screenshot with optional scaling and cropping.

Parameters

Name	Type	Description
`maxWidth`	`number`	Maximum width to scale to
`maxHeight`	`number`	Maximum height to scale to
`quality`	`number`	JPEG quality (1-100)
`cropX1?`	`number`	Crop region left
`cropY1?`	`number`	Crop region top
`cropX2?`	`number`	Crop region right
`cropY2?`	`number`	Crop region bottom

Returns

{screenshot, compressedWidth, compressedHeight, originalWidth, originalHeight}Screenshot data as base64 string with dimensions

Examples

Full screenshot

TypeScript

const result = await agent.actions.screenshot(1080, 1920, 80);

Cropped screenshot

TypeScript

const result = await agent.actions.screenshot(500, 500, 90, 100, 100, 600, 600);

`nodeAction()`

TypeScript

nodeAction(node: AndroidNode | object, actionInt: number, data?: object, fieldsToIgnore?: string[]): Promise<{actionPerformed: boolean}>

Performs an accessibility action on a node.

Parameters

Name	Type	Description
`node`	`AndroidNode \| object`	The node to perform action on
`actionInt`	`number`	Action constant (use agent.constants.ACTION_*)
`data?`	`object`	Additional action data
`fieldsToIgnore?`	`string[]`	Node fields to ignore when matching

Returns

{actionPerformed: boolean}Whether the action was successfully performed

Examples

Using node.performAction (preferred)

TypeScript

const screen = await agent.actions.screenContent();
const button = screen.findTextOne("Submit");

if (button) {
  const result = await button.performAction(agent.constants.ACTION_CLICK);
  console.log("Clicked:", result.actionPerformed);
}

Using agent.actions.nodeAction (legacy)

TypeScript

const result = await agent.actions.nodeAction(
  button,
  agent.constants.ACTION_CLICK
);

`showNotification()`

TypeScript

showNotification(title: string, message: string): Promise<void>

Shows a system notification.

Parameters

Name	Type	Description
`title`	`string`	Notification title
`message`	`string`	Notification message

Returns

Promise<void>Resolves when notification is shown

Examples

TypeScript

await agent.actions.showNotification("Task Complete", "Your automation has finished.");

`isDeviceLocked()`

TypeScript

isDeviceLocked(): Promise<boolean>

Whether the device is locked right now — the screen is off or the lock screen is showing. Unlike isScreenLockSet (is a PIN / pattern / password set up?), this is the current state. Same check as agent.actions.adb.isDeviceLocked(), which never needed an ADB connection. Requires an app version newer than 3.8.0 (versionCode 30022 or later).

Returns

Promise<boolean>true when the screen is off or the keyguard is showing

Examples

TypeScript

if (await agent.actions.isDeviceLocked()) {
  console.log("Device is locked — wake / unlock before interacting");
}

`isScreenLockSet()`

TypeScript

isScreenLockSet(): Promise<boolean | null>

Whether a secure screen lock (PIN, pattern or password) is set up on the device. This reports the lock setting, not whether the screen is locked right now. Requires an app version newer than 3.7.0 (versionCode 30021 or later).

Returns

Promise<boolean | null>true if a PIN / pattern / password is set, false if not, null if it could not be determined

Examples

TypeScript

if (await agent.actions.isScreenLockSet()) {
  console.log("Device has a PIN / pattern / password lock");
}

`speak()`

TypeScript

speak(text: string, delayMs?: number): Promise<void>

Reads text aloud using the device's text-to-speech (TextToSpeech) engine.

Parameters

Name	Type	Description
`text`	`string`	The text to speak
`delayMs?`	`number`	Delay in milliseconds before speaking

Returns

Promise<void>Resolves once the utterance has been queued to the TTS engine

Examples

TypeScript

await agent.actions.speak("Automation finished");

Speak after a delay

TypeScript

await agent.actions.speak("Heads up", 1000);

`displayDialog()`

TypeScript

displayDialog(text: string, heading: string, overlayOtherApps?: boolean, delayMs?: number): Promise<void>

Displays a simple dialog to the device user, reusing the in-app-message system (the same overlay/dialog shown by push notifications). With overlayOtherApps=true the dialog is drawn over whatever app is in the foreground (requires the accessibility service); otherwise it is shown the next time the app is opened.

Parameters

Name	Type	Description
`text`	`string`	The dialog body text
`heading`	`string`	Heading shown above the body (pass an empty string for none)
`overlayOtherApps?`	`boolean`	Show immediately as an overlay over other apps (default false)
`delayMs?`	`number`	Delay in milliseconds before showing the dialog

Returns

Promise<void>Resolves once the dialog has been dispatched

Examples

TypeScript

await agent.actions.displayDialog("Please keep the screen on", "Notice");

Overlay over the current app

TypeScript

await agent.actions.displayDialog("Do not touch", "Automation running", true);