Configure

Start from the symptom

What works as is

With no configuration, Marionette recognizes Flutter's own interactive widgets, their subclasses (such as the button ElevatedButton.icon returns) and their generic variants:

ButtonStyleButton (Elevated, Filled, Outlined and Text buttons), IconButton, FloatingActionButton, TextField, TextFormField, Checkbox, Radio, Switch, Slider, DropdownButton, PopupMenuButton, their list-tile forms, GestureDetector and InkWell.

Text is read from Text, RichText, EditableText and the text fields. A field’s labelText or hintText is reported as its label, so a field without a key still has a name.

The configuration

MarionetteConfiguration is the optional argument of MarionetteBinding.ensureInitialized. Every field is optional, and your callbacks run after the built-in checks: they extend the defaults, they do not replace them.

Interactive widgets

A screen has hundreds of widgets; the agent gets the few it can act on. Your design system’s widgets are not on the built-in list, so mark them. The callback gets the Element, so one switch over element.widget describes a whole design system.

MarionetteConfiguration(
  isInteractiveElement: (element) => switch (element.widget) {
    MyButton() || MyTextField() => true,
    MyTile(:final onTap) => onTap != null,
    _ => false,
  },
)

• A pattern such as MySelect() matches MySelect<String> and any type argument added later.
• MyButton() matches every subclass, such as MyPrimaryButton.
• Read fields for per-instance decisions: a MyTile is interactive only when it has an onTap.
• Use the element as a BuildContext when placement matters, e.g. findAncestorWidgetOfExactType.

Text

extractText does two jobs. Widgets with text appear in the listing with that text, so the agent knows what each one shows. And tap and scroll_to can then match them by label. enter_text targets fields by key or identifier, or the field focused with a tap.

MarionetteConfiguration(
  extractText: (element) => switch (element.widget) {
    MyText(:final data) => data,
    MyTextField(:final labelText) => labelText,
    _ => null,
  },
)

Semantics

Text.rich and RichText are read without help. For text that never reaches a Text widget, such as CustomPaint, TextPainter or WidgetSpan content, use the annotation screen readers already rely on.

// The screen reader says "Volume, 70 percent";
// the agent sees  Type: Semantics, Text: "Volume: 70%".
Semantics(
  label: 'Volume',
  value: '70%',
  child: CustomPaint(painter: _VolumeBarPainter(level: 0.7)),
)

label and value are for reading only: the matcher ignores them, so a Semantics(label: "Submit") never steals a tap from the button inside it. To target a widget without a key, use identifier, which the matcher does read and screen readers never announce.

// No ValueKey, but a stable target: tap(identifier: "submit_button").
Semantics(
  identifier: 'submit_button',
  child: ElevatedButton(onPressed: _submit, child: const Text('Submit')),
)

Text(semanticsIdentifier:) works the same way. A TextSpan.semanticsIdentifier does not: it lives below the widget tree, so wrap that run of text in its own Semantics(identifier:) instead.

Traversal

To list the screen, Marionette walks the widget tree from the root down. shouldStopTraversalAtElement tells it not to look inside a widget: the widget itself is still listed, everything below it is not.

Logs

get_logs returns what the app logged since it started or since the last hot reload: the quickest proof that a request fired or an exception was thrown. Pick the adapter for how your app logs.

flutter pub add marionette_logging

MarionetteBinding.ensureInitialized(
  MarionetteConfiguration(logCollector: LoggingLogCollector()),
);
Logger.root.level = Level.ALL;

flutter pub add marionette_logger

final logCollector = LoggerLogCollector();
MarionetteBinding.ensureInitialized(
  MarionetteConfiguration(logCollector: logCollector),
);
// The collector is also a LogOutput, so one object serves both sides.
final logger = Logger(output: MultiOutput([ConsoleOutput(), logCollector]));

final collector = PrintLogCollector();
MarionetteBinding.ensureInitialized(
  MarionetteConfiguration(logCollector: collector),
);
myLogger.onLog((message) => collector.addLog(message));

Device config

set_device_config overrides text scale, bold text and light or dark mode in the running app. It is the one tool that needs a widget in your tree, because Marionette never inserts one behind your back. Hot restart after adding it: main() does not re-run on a hot reload.

void main() {
  if (!kReleaseMode) {
    MarionetteBinding.ensureInitialized();
    runApp(const MarionetteDeviceConfig(child: MyApp()));
  } else {
    WidgetsFlutterBinding.ensureInitialized();
    runApp(const MyApp());
  }
}

!kReleaseMode is a compile-time constant, so neither the binding nor the widget reaches a release build. Without a binding the widget builds its child unchanged. When reading results, note that some Material widgets clamp their own text scaling, and scales far above 3.0 mostly produce overflow errors, not signal.

Custom extensions

Give the agent actions the generic tools cannot express: go to a route by name, seed test data, flip a feature flag. An extension that declares an inputSchema becomes a named tool with validated arguments, so a bad value is rejected before it reaches your app.

registerMarionetteExtension(
  name: 'appNavigation.goToPage',
  description: 'Navigates to a page by name.',
  inputSchema: ExtensionInputSchema(
    properties: {
      'page': ExtensionParam.string(
        description: 'The page to navigate to.',
        enumValues: ['home', 'profile', 'settings'],
      ),
    },
    required: ['page'],
  ),
  callback: (params) async {
    final page = params['page'];
    if (page == null) {
      return MarionetteExtensionResult.invalidParams('Missing page');
    }
    navigateTo(page); // your app's own navigation
    return MarionetteExtensionResult.success({'page': page});
  },
);

Tool names are reduced to [a-z0-9_-], the set every LLM API accepts. Parameters are flat scalars, because arguments travel over the VM service as strings. An extension without a schema still works through call_custom_extension, and list_custom_extensions shows them all.

A complete main.dart

import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:logging/logging.dart';
import 'package:marionette_flutter/marionette_flutter.dart';
import 'package:marionette_logging/marionette_logging.dart';

void main() {
  if (!kReleaseMode) {
    MarionetteBinding.ensureInitialized(
      MarionetteConfiguration(
        // 1. Your interactive widgets.
        isInteractiveElement: (element) => switch (element.widget) {
          MyButton() || MyTextField() => true,
          _ => false,
        },
        // 2. Their text, so the agent can match by label.
        extractText: (element) => switch (element.widget) {
          MyText(:final data) => data,
          _ => null,
        },
        // 3. Logs for get_logs.
        logCollector: LoggingLogCollector(),
        // 4. A report for every run.
        enableSessionReports: true,
      ),
    );
    Logger.root.level = Level.ALL;
    // 5. set_device_config, opted in with one widget.
    runApp(const MarionetteDeviceConfig(child: MyApp()));
  } else {
    runApp(const MyApp());
  }
}