How it works

Agent

calls tap(key: "save")

MCP or shell

Bridge

translates it into a VM service call

VM service

Binding

runs ext.flutter.marionette.tap in the app

gesture

App

gets a real pointer event

MarionetteBinding registers a set of VM service extensions in your app. On connect, the bridge checks that the app runs the same Marionette version. From then on every tool call becomes one extension call: the app performs the action and answers with a short result, which goes back to the agent.

Gestures are dispatched as pointer events, so the app reacts the way it would to a finger: hit tests, gesture arenas and focus all apply. Nothing in your widgets changes.

How the agent sees a screen

get_interactive_elements walks the widget tree and returns a flat list in tree order: every widget a user can act on, every widget with text, and every Semantics with a label or value. Each entry carries its type, key, text, Semantics identifier and bounds, so the agent can name a target and knows where it is.

Below is one real screen of the example app and the list the agent received for it. Point at an entry to find it on the screen; switch to the full payload to see what compaction leaves out.

The Custom Widgets screen of the example app

take_screenshots is the other way to look. It costs far more context than the list, so agents use it to check layout or to keep evidence, and the list to act.

Choosing a target

Every gesture and text tool matches its target the same way. From the most to the least stable:

Selector	Survives	Use it when
`key`	copy changes, translation, refactors	the widget has a `ValueKey<String>`. Prefer it.
`identifier`	the same, and it is never read aloud	no key fits: wrap it in `Semantics(identifier:)`.
`text`	nothing that edits or translates the label	a quick look, or a one-off run.
`type`	only screens with one widget of that type	the screen has one `PageView` or one `Switch`.
`coordinates`	nothing: any layout or font-scale change	a diagnostic. If you need them, the widget needs a key.

The first match in the tree wins. When the same key repeats in identical subtrees, such as grid cells or repeated cards, add ancestor_keys: the keys of the wrappers, outermost first. Each one is looked up inside the previous one, and a key with no element fails the call by name instead of widening the search.

tap

{ "key": "cell.joinButton", "ancestor_keys": ["session_2", "grid.cell_3"] }

Compact by default

The list never carries style, decoration or colour objects: only primitive values. On top of that, the default CompactionMode.compact drops rendering details no tool reads, drops a Text element’s duplicated data, rounds bounds to whole logical pixels, and reports visible only when an element is hidden. On the example app that makes the payload about 65% smaller.

Nothing it drops is read by tap, enter_text, scroll_to or swipe. When the agent needs the full dump, it passes compaction: "none" (--compaction=none in the CLI), or the app sets compaction: CompactionMode.none as its default.

Debug and profile only

The VM service exists in debug and profile builds, never in release. That is a design choice, not a gap: gate the binding on kDebugMode, or on !kReleaseMode to include profile builds. Both are compile-time constants, so the release binary contains no Marionette code.

Hot reload keeps the app’s state and does not re-run main(). A change to main(), to a global singleton or to the shape of the state needs hot_restart, which works when the app runs under flutter run.

One binding per process

Flutter allows one WidgetsBinding per process, and MarionetteBinding is one. If another binding got there first, ensureInitialized throws and names it. Two cases come up often.

Tests that call `main()`

kDebugMode is true under flutter test, so the test binding and Marionette collide. Skip Marionette under test, or give tests their own entrypoint that never initializes it.

lib/main.dart

import 'dart:io' show Platform;

final isFlutterTest = Platform.environment.containsKey('FLUTTER_TEST');
if (kDebugMode && !isFlutterTest) {
  MarionetteBinding.ensureInitialized();
} else {
  WidgetsFlutterBinding.ensureInitialized();
}

Plugins with their own binding, such as Sentry

SentryFlutter.init installs a binding before it calls appRunner. Initialized inside appRunner, Marionette throws inside Sentry’s error zone and the app hangs on the splash screen with no error at all. Initialize Marionette first; Sentry reuses the binding it finds.

lib/main.dart

void main() {
  if (kDebugMode) {
    MarionetteBinding.ensureInitialized(); // before Sentry, not inside appRunner
  } else {
    WidgetsFlutterBinding.ensureInitialized();
  }
  SentryFlutter.init(
    (options) => options.dsn = _sentryDsn,
    appRunner: () => runApp(const MyApp()),
  );
}

MCP server or CLI

Both bridges drive the same app. Pick the one that fits your agent and the way you work; most coding agents can use either.

	MCP server	CLI
Package	`marionette_mcp`	`marionette_cli`, the `marionette` command
Agent needs	MCP support	a shell, and the `help-ai` reference
Connection	`connect` once, then calls	`--uri` per call, or `register` a named instance
Custom extensions	schema-bearing ones become named tools	not available
Video	—	`record-video`, with ffmpeg

Both need the same setup in the app, and both write the same session reports.

When not to use it

A regression suite in CI. Gestures are a best-effort simulation of a user, not deterministic instrumentation. For tests that pass or fail the same way every time, use Patrol.
A release build. The VM service does not exist there.
Performance work. Marionette can trigger the interaction to profile; measuring it is the job of Flutter DevTools.
Writing the app: dependencies, analysis and runtime errors belong to the Dart and Flutter MCP server. Use both.