Native lane
Some of what a user sees is drawn by the platform, not by Flutter. The native lane drives it through the device’s own UI automation, next to the Flutter tools.
When to switch lanes
The Flutter tools (the Flutter lane) see the widget tree. The native_* tools (the native lane) talk to UIAutomator2 on Android and WebDriverAgent on the iOS Simulator, and see everything on screen: permission dialogs, system alerts, pickers and share sheets, WebView content, native controls in a platform view, native screens and the notification shade.
Prefer the Flutter lane for widgets in the app. Switch when the target is not in the Flutter tree, and switch back once the app is in front again. While system UI covers the app,
get_interactive_elements starts with a warning that the app is not in front: the widgets it lists are underneath and a tap on them does nothing.
What each lane sees
| UI | Flutter lane | Native lane |
|---|---|---|
| Widgets of the app | Yes | Prefer the Flutter lane |
| Permission dialogs, system alerts | No | Read, tap |
| Photo and file pickers, share sheet | No | Read, tap |
| WebView content | No | Fields, buttons, links |
| Native controls in a platform view | No | Read, tap, type |
| Native screens | No | Read, tap, type |
| Notifications | No | In the system list |
| Camera preview, map, video | No | Only controls around |
| Software keyboard | No | iOS: keys; Android: hidden |
Set up the device
The app needs nothing new: the native lane does not go through MarionetteBinding. The device does. The native lane runs on macOS and Linux hosts; Windows is not supported. To run the MCP server without the native tools and their instructions, start it with
--no-native (marionette_mcp --no-native or marionette mcp --no-native).
Android
adb on PATH and a device or emulator that adb devices lists as
device. On the first connect Marionette downloads the pinned UIAutomator2 APKs, checks their SHA-256 and installs them; later connects reuse them.
MARIONETTE_UIA2_SERVER_APK and MARIONETTE_UIA2_TEST_APK point at local copies,
MARIONETTE_UIA2_PORT fixes the local port.
iOS Simulator
macOS with Xcode and a booted simulator; physical iPhones are not supported yet. Marionette drives one iOS Simulator at a time: WebDriverAgent listens on port 8100. It is built once, for the simulator runtime you use, and Marionette is pointed at the build. Marionette also finds a build copied to
marionette_mcp/wda/ under $XDG_CACHE_HOME (default ~/.cache), and reuses a WebDriverAgent already running for the simulator.
git clone https://github.com/appium/WebDriverAgent.git
cd WebDriverAgent
xcodebuild build-for-testing \
-project WebDriverAgent.xcodeproj \
-scheme WebDriverAgentRunner \
-destination 'platform=iOS Simulator,id=<UDID>'
export MARIONETTE_WDA_XCTESTRUN=/path/to/WebDriverAgentRunner_*.xctestrun
Connect
native_connect with platform: android or ios, plus serial
or udid when several devices are attached. It is separate from connect: a run usually uses both. Call it before the flow opens system UI: starting WebDriverAgent can dismiss an alert already on screen. If a native call later says the device-side server or session is gone, call
native_connect again; native_disconnect ends it.
In the CLI, marionette native-connect starts the device-side server detached from the CLI, so later
native-* commands reuse it without setup, and native-disconnect stops it. --native-device <name>
names a connection to drive several devices at once; native-sessions lists them.
$ marionette native-connect --platform android
$ marionette native-wait-for --text "Allow" --timeout-ms 5000
$ marionette native-tap --text "Allow"
$ marionette native-get-elements --clickable-only
$ marionette native-take-screenshot -o dialog.png
$ marionette native-disconnect
Prompts
Name the system UI the flow goes through; the agent switches lanes on its own. Add the ws://
URI and the device, as for any run.
Tap "Add photo" on Profile. When the system asks for photo access, allow it with the native tools, then check that the picker opened.
The dialog answered natively, and the picker seen behind it.
On Documents, tap "Upload", pick report.pdf in the system file picker, and check that the file name shows up in the app.
A file chosen in system UI, verified back in Flutter.
Open Terms, scroll the WebView to the checkbox, check it, tap "Accept", and confirm the app moves on to Home.
WebView content driven through the native tree.
Trigger the order notification, send the app to the background, open the notification from the system list, and check that Order details opens.
The notification read and opened from the shade.
How the agent should work
-
Look first:
native_get_elementslists type, text, id, bounds and state, plusforegroundApp, the app in front. A text field’s typed content is itsvalue; itstextstays the label. -
Select by
text,textContains,id,typeandindex;native_enter_texttakes the field’slabel. An ambiguous selector fails and lists the candidates, instead of guessing. -
Native coordinates (
x/y, bounds, swipe points) are screen pixels on every platform. Never reuse Flutter coordinates in the native lane, or the other way round. -
native_take_screenshotfits the image in 2000×2000, liketake_screenshots. When it shrinks it, the result gives the factor: multiply a point read off the image by it to get nativex/y. -
Wait with
native_wait_for(visibleorgone) for a dialog or the next screen, not with fixed delays. Wait for an element of that screen, not for a package name. -
Scroll with
native_scroll: with a target it stops once the target is visible or the list ends;withinscrolls a nested list or a picker wheel.native_swipeis a raw gesture that does not wait for the content to settle. -
Hide the keyboard with
native_hide_keyboardwhen it covers the next target.native_enter_textwithout a selector types into the focused field. - A disabled element is refused, not touched. On iOS the app’s elements stay listed under a sheet or an alert, and not all of them are disabled: act on what is in front first.
-
Read before you look:
native_get_elementscosts a few hundred tokens on a system screen, a screenshot about 1.5k.native_take_screenshotshows the whole screen, system UI included, so use it for what the tree does not show, such as banners, toasts or a camera preview.
Permission dialogs
Answer the dialog the way a user does: native_wait_for the button, then native_tap
it, for example text: "Allow". The system shows a dialog only while the permission is undecided. To see it again, reset the permission outside Marionette:
adb shell pm revoke <package> <permission>
xcrun simctl privacy <udid> reset all <bundle id>
Notifications
native_notifications opens and closes the notification shade on Android and the Notification Center on iOS. Read the notifications with
native_get_elements, open one with native_tap (on iOS, a swipe — see below).
-
On iOS, send the app to the background first (
native_press_keyhome): opened over the app, the Notification Center is not in the tree. Open it after the banner is gone: a banner on screen can take the swipe. - iOS groups an app’s notifications: the first tap on a group expands it. Open a notification by swiping it right across its full width; a tap does not open it.
-
Android heads-up banners, and toasts on recent Android versions and on iOS, appear only on
native_take_screenshot.
Every native tool
| MCP tool | CLI command | What it does |
|---|---|---|
native_connect |
native-connect |
Start the device-side server and a session. |
native_disconnect |
native-disconnect |
End the session; stop the server it started. |
native_get_elements |
native-get-elements |
List what is on screen, with state. |
native_wait_for |
native-wait-for |
Wait until an element is visible or gone. |
native_tap |
native-tap |
By selector, or at x/y in screen pixels. |
native_enter_text |
native-enter-text |
Type into a field, or the focused one. |
native_scroll |
native-scroll |
In a direction, to a target, or within a list. |
native_swipe |
native-swipe |
From one point to another: drags, edge gestures. |
native_press_key |
native-press-key |
back (Android), home, enter and more. |
native_hide_keyboard |
native-hide-keyboard |
Hide the software keyboard. |
native_take_screenshot |
native-take-screenshot |
The whole screen; inline: false saves it. |
native_notifications |
native-notifications |
Open or close the notification list. |
| — | native-sessions |
List the CLI’s device connections. |
In session reports
Native calls go into the same steps.md as the Flutter ones, once connect has opened a session, so a report covers both lanes. Text typed with
native_enter_text is logged as its length, never as the text. native_take_screenshot(inline: false)
saves into screenshots/ like take_screenshots. See Reports.
Start from the symptom
| Symptom | Fix |
|---|---|
| A Flutter tap does nothing while a dialog is open | System UI covers the app: use the native tools. |
native_connect on Android: no device or no adb |
Start an emulator; put platform-tools on PATH. |
native_connect on Android: the device is locked |
Unlock it once after a restart, then connect again. |
native_connect on iOS: no WebDriverAgent build |
Build it once: Set up the device. |
native_connect on iOS: port 8100 is taken |
One iOS Simulator at a time: stop what lsof -nP -iTCP:8100 -sTCP:LISTEN shows. |
| A native call says several elements match | Add index, or a more specific type or id. |
| A native call says the session or server is gone | native_connect again; one client per device. |
native_wait_for misses an element you can see |
It is below the screen edge: scroll it into view. |
native_hide_keyboard refuses on an emulator |
Gboard's floating toolbar: unfocus the field in the app. |
A WebView checkbox shows no checked state (Android) |
Android WebView does not report it: read the page instead. |
| A tap in an iOS WebView hits the wrong element | The keyboard scrolled it: tap by screenshot coordinates. |
| An iOS notification does not open on tap | Swipe it right across its full width with native_swipe. |