Skip to content

Native lane

Some of what a user sees is drawn by the platform, not by Flutter. The native lane drives it through the device’s own UI automation, next to the Flutter tools.

When to switch lanes

The Flutter tools (the Flutter lane) see the widget tree. The native_* tools (the native lane) talk to UIAutomator2 on Android and WebDriverAgent on the iOS Simulator, and see everything on screen: permission dialogs, system alerts, pickers and share sheets, WebView content, native controls in a platform view, native screens and the notification shade.

Prefer the Flutter lane for widgets in the app. Switch when the target is not in the Flutter tree, and switch back once the app is in front again. While system UI covers the app, get_interactive_elements starts with a warning that the app is not in front: the widgets it lists are underneath and a tap on them does nothing.

What each lane sees

UIFlutter laneNative lane
Widgets of the app Yes Prefer the Flutter lane
Permission dialogs, system alerts No Read, tap
Photo and file pickers, share sheet No Read, tap
WebView content No Fields, buttons, links
Native controls in a platform view No Read, tap, type
Native screens No Read, tap, type
Notifications No In the system list
Camera preview, map, video No Only controls around
Software keyboard No iOS: keys; Android: hidden

Set up the device

The app needs nothing new: the native lane does not go through MarionetteBinding. The device does. The native lane runs on macOS and Linux hosts; Windows is not supported. To run the MCP server without the native tools and their instructions, start it with --no-native (marionette_mcp --no-native or marionette mcp --no-native).

Android

adb on PATH and a device or emulator that adb devices lists as device. On the first connect Marionette downloads the pinned UIAutomator2 APKs, checks their SHA-256 and installs them; later connects reuse them. MARIONETTE_UIA2_SERVER_APK and MARIONETTE_UIA2_TEST_APK point at local copies, MARIONETTE_UIA2_PORT fixes the local port.

iOS Simulator

macOS with Xcode and a booted simulator; physical iPhones are not supported yet. Marionette drives one iOS Simulator at a time: WebDriverAgent listens on port 8100. It is built once, for the simulator runtime you use, and Marionette is pointed at the build. Marionette also finds a build copied to marionette_mcp/wda/ under $XDG_CACHE_HOME (default ~/.cache), and reuses a WebDriverAgent already running for the simulator.

terminal
git clone https://github.com/appium/WebDriverAgent.git
cd WebDriverAgent
xcodebuild build-for-testing \
  -project WebDriverAgent.xcodeproj \
  -scheme WebDriverAgentRunner \
  -destination 'platform=iOS Simulator,id=<UDID>'
export MARIONETTE_WDA_XCTESTRUN=/path/to/WebDriverAgentRunner_*.xctestrun

Connect

native_connect with platform: android or ios, plus serial or udid when several devices are attached. It is separate from connect: a run usually uses both. Call it before the flow opens system UI: starting WebDriverAgent can dismiss an alert already on screen. If a native call later says the device-side server or session is gone, call native_connect again; native_disconnect ends it.

In the CLI, marionette native-connect starts the device-side server detached from the CLI, so later native-* commands reuse it without setup, and native-disconnect stops it. --native-device <name> names a connection to drive several devices at once; native-sessions lists them.

a permission dialog from the CLI
$ marionette native-connect --platform android
$ marionette native-wait-for --text "Allow" --timeout-ms 5000
$ marionette native-tap --text "Allow"
$ marionette native-get-elements --clickable-only
$ marionette native-take-screenshot -o dialog.png
$ marionette native-disconnect

Prompts

Name the system UI the flow goes through; the agent switches lanes on its own. Add the ws:// URI and the device, as for any run.

Allow a permission
Tap "Add photo" on Profile. When the system asks for photo access, allow it with the native tools, then check that the picker opened.

The dialog answered natively, and the picker seen behind it.

Pick a file
On Documents, tap "Upload", pick report.pdf in the system file picker, and check that the file name shows up in the app.

A file chosen in system UI, verified back in Flutter.

Fill a WebView
Open Terms, scroll the WebView to the checkbox, check it, tap "Accept", and confirm the app moves on to Home.

WebView content driven through the native tree.

Open a notification
Trigger the order notification, send the app to the background, open the notification from the system list, and check that Order details opens.

The notification read and opened from the shade.

How the agent should work

  • Look first: native_get_elements lists type, text, id, bounds and state, plus foregroundApp, the app in front. A text field’s typed content is its value; its text stays the label.
  • Select by text, textContains, id, type and index; native_enter_text takes the field’s label. An ambiguous selector fails and lists the candidates, instead of guessing.
  • Native coordinates (x/y, bounds, swipe points) are screen pixels on every platform. Never reuse Flutter coordinates in the native lane, or the other way round.
  • native_take_screenshot fits the image in 2000×2000, like take_screenshots. When it shrinks it, the result gives the factor: multiply a point read off the image by it to get native x/y.
  • Wait with native_wait_for (visible or gone) for a dialog or the next screen, not with fixed delays. Wait for an element of that screen, not for a package name.
  • Scroll with native_scroll: with a target it stops once the target is visible or the list ends; within scrolls a nested list or a picker wheel. native_swipe is a raw gesture that does not wait for the content to settle.
  • Hide the keyboard with native_hide_keyboard when it covers the next target. native_enter_text without a selector types into the focused field.
  • A disabled element is refused, not touched. On iOS the app’s elements stay listed under a sheet or an alert, and not all of them are disabled: act on what is in front first.
  • Read before you look: native_get_elements costs a few hundred tokens on a system screen, a screenshot about 1.5k. native_take_screenshot shows the whole screen, system UI included, so use it for what the tree does not show, such as banners, toasts or a camera preview.

Permission dialogs

Answer the dialog the way a user does: native_wait_for the button, then native_tap it, for example text: "Allow". The system shows a dialog only while the permission is undecided. To see it again, reset the permission outside Marionette:

terminal
adb shell pm revoke <package> <permission>
xcrun simctl privacy <udid> reset all <bundle id>

Notifications

native_notifications opens and closes the notification shade on Android and the Notification Center on iOS. Read the notifications with native_get_elements, open one with native_tap (on iOS, a swipe — see below).

  • On iOS, send the app to the background first (native_press_key home): opened over the app, the Notification Center is not in the tree. Open it after the banner is gone: a banner on screen can take the swipe.
  • iOS groups an app’s notifications: the first tap on a group expands it. Open a notification by swiping it right across its full width; a tap does not open it.
  • Android heads-up banners, and toasts on recent Android versions and on iOS, appear only on native_take_screenshot.

Every native tool

MCP toolCLI commandWhat it does
native_connect native-connect Start the device-side server and a session.
native_disconnect native-disconnect End the session; stop the server it started.
native_get_elements native-get-elements List what is on screen, with state.
native_wait_for native-wait-for Wait until an element is visible or gone.
native_tap native-tap By selector, or at x/y in screen pixels.
native_enter_text native-enter-text Type into a field, or the focused one.
native_scroll native-scroll In a direction, to a target, or within a list.
native_swipe native-swipe From one point to another: drags, edge gestures.
native_press_key native-press-key back (Android), home, enter and more.
native_hide_keyboard native-hide-keyboard Hide the software keyboard.
native_take_screenshot native-take-screenshot The whole screen; inline: false saves it.
native_notifications native-notifications Open or close the notification list.
— native-sessions List the CLI’s device connections.

In session reports

Native calls go into the same steps.md as the Flutter ones, once connect has opened a session, so a report covers both lanes. Text typed with native_enter_text is logged as its length, never as the text. native_take_screenshot(inline: false) saves into screenshots/ like take_screenshots. See Reports.

Start from the symptom

SymptomFix
A Flutter tap does nothing while a dialog is open System UI covers the app: use the native tools.
native_connect on Android: no device or no adb Start an emulator; put platform-tools on PATH.
native_connect on Android: the device is locked Unlock it once after a restart, then connect again.
native_connect on iOS: no WebDriverAgent build Build it once: Set up the device.
native_connect on iOS: port 8100 is taken One iOS Simulator at a time: stop what lsof -nP -iTCP:8100 -sTCP:LISTEN shows.
A native call says several elements match Add index, or a more specific type or id.
A native call says the session or server is gone native_connect again; one client per device.
native_wait_for misses an element you can see It is below the screen edge: scroll it into view.
native_hide_keyboard refuses on an emulator Gboard's floating toolbar: unfocus the field in the app.
A WebView checkbox shows no checked state (Android) Android WebView does not report it: read the page instead.
A tap in an iOS WebView hits the wrong element The keyboard scrolled it: tap by screenshot coordinates.
An iOS notification does not open on tap Swipe it right across its full width with native_swipe.