Skip to content

Reports

An agent’s summary is a claim. A session report is a claim with its evidence next to it: the server logs every call, and the agent writes the report from that log.

Turn them on

Session reports are off by default; then nothing is written to disk and screenshots only come back inline. One flag turns them on:

lib/main.dart
MarionetteBinding.ensureInitialized(
  const MarionetteConfiguration(enableSessionReports: true),
);

One run, one folder

Every connect opens a fresh folder, named after the session_title the agent passes and the time. A run is never resumed: an interrupted one is abandoned, and the next connect starts over, so a report can never be overwritten by a later, unrelated run.

project root
.marionette/                         # writes its own .gitignore
  sessions/
    profile-email-validation-20261006T1326/
      steps.md        # appended by the server, one line per call
      report.md       # written once, at the end, by the agent
      screenshots/
        01.png        # take_screenshots(inline: false)
  • Where: the session_dir argument of connect (--session-dir in the CLI), else the MARIONETTE_SESSION_DIR variable, else the working directory.
  • The 20 most recently used folders are kept. A folder with a report.md is never pruned.
  • If the folder cannot be created, the connection still works, just without a record.

steps.md

The server appends one line per tool call as it happens: the tool, a short selector and the outcome. The agent never writes to it, so a step cannot be skipped or improved after the fact. These are the real lines of the run on the home page:

steps.md
- [13:26:45] connect uri=ws://127.0.0.1:36383 -> Successfully connected to app at ws://127.0.0.1:36383
- [13:26:47] hot_reload -> Hot reload completed successfully
- [13:26:47] tap text=Profile -> Tapped element matching: {text: Profile}
- [13:26:50] get_interactive_elements -> ok
- [13:26:50] enter_text key=name_field input_len=12 -> Entered text into element matching: {key: name_field}
- [13:26:52] enter_text key=email_field input_len=11 -> Entered text into element matching: {key: email_field}
- [13:26:55] take_screenshots -> 1 screenshot(s) captured
- [13:26:55] get_interactive_elements -> ok
- [13:26:56] enter_text key=email_field input_len=15 -> Entered text into element matching: {key: email_field}
- [13:26:58] enter_text key=bio_field input_len=15 -> Entered text into element matching: {key: bio_field}
- [13:27:00] tap text=Submit -> Tapped element matching: {text: Submit}
- [13:27:01] get_interactive_elements -> ok
- [13:27:02] take_screenshots -> Saved 1 screenshot(s): screenshots/01.png
- [13:27:02] get_logs -> ok
- [13:27:03] disconnect -> Successfully disconnected from app
  • No payloads: a call that returns app data, such as get_interactive_elements or get_logs, is logged as ok.
  • Typed text is logged as its length, input_len=12. For a field whose key looks like a password, PIN, token, secret or CVV, even the length is withheld.
  • A VM service URI is cut to ws://host:port: its path holds a live debug-access token.
  • A crash in the app is logged as the exception and a few stack frames, not the raw error payload.

For evidence, the agent calls take_screenshots(inline: false): the image goes to screenshots/ and the agent gets its path instead of the pixels, which saves context.

report.md

The agent writes report.md once, at the end, or when it stops early and says why. disconnect reminds it, with the folder’s path. The format is a contract the drive Agent Skill teaches:

  • The Tested: line is counted from steps.md, never from memory.
  • Every finding cites a step, a log line or a screenshot. Without evidence it is a [suspicion], and says so.
  • Under six lines per finding: a tag, a summary, a repro path, the evidence.
  • A status icon on the headline and each finding: ✅ clean, ⚠️ suspicion, 🐛 bug, ➖ not covered.
  • Written in the language of the prompt; tool names, paths and keys stay as they are.
report.md
✅ Marionette report — no issues · Profile email validation
Tested: hot reload, Profile form — name, malformed email, valid email, bio, submit (15 steps, 1 screenshot)
Checks: "ada@example" shows "Enter a valid email" and Submit stays disabled (steps 6–8); a valid email and a bio enable Submit, and submitting shows "Form submitted" (steps 9–12, screenshots/01.png); get_logs collected nothing, so no errors were logged (step 14)
report.md · format example
🐛 Marionette report — 2 findings · User Profile
Tested: profile view, edit form, save flow (14 steps, 3 screenshots)

1. 🐛 [bug] Date of birth accepts future dates — no validation
   Repro: Profile → Edit → DOB = 2099-01-01 → Save
   Evidence: step 9, log "saved dob=2099-01-01"

2. 🐛 [bug] Save does not persist — changes lost after restart
   Repro: Profile → Edit → Name = "X" → Save → hot_restart → Profile
   Evidence: step 14, screenshots/02-after-restart.png

The closing message

The chat gets a short message, not a second copy of the report. Its verdict is one of four words, never translated: PASS, SUSPECT, FAIL or BLOCKED. BLOCKED wins when the run could not finish, even if it also found a bug.

template
Marionette · <flow> — <VERDICT>: <reason, a few words>
✓ <what worked, comma-separated>
✗ [<type>] <finding, one line>        (up to 5)
∅ Not covered: <what> — <why>
→ <session>/report.md · <N> steps, <M> screenshots
the recorded run
Marionette · Profile email validation — PASS: validation and submit work
✓ malformed email rejected, Submit disabled until the form is valid, form submits
→ .marionette/sessions/profile-email-validation-20261006T1326/report.md · 15 steps, 1 screenshot
format example
Marionette · User Profile — FAIL: 2 bugs
✓ Profile view, edit form
✗ [bug] Date of birth accepts future dates — no validation
✗ [bug] Save does not persist — changes lost after restart
→ .marionette/sessions/user-profile-20260922T1412/report.md · 14 steps, 3 screenshots
format example
Marionette · Settings — SUSPECT: theme toggle may not persist
✓ Notification toggles, language picker
✗ [suspicion] Dark mode toggle reverts to light after hot_restart
∅ Not covered: account deletion, data export — out of scope for this run
→ .marionette/sessions/settings-20260922T1601/report.md · 9 steps, 2 screenshots
format example
Marionette · Password reset — BLOCKED: reset email never arrives
✓ Request form, validation messages
✗ [blocker] "Send reset link" succeeds but no email is logged after 3 attempts
∅ Not covered: setting a new password — the flow cannot get past the email
→ .marionette/sessions/password-reset-20260922T1601/report.md · 11 steps, 1 screenshot

Sharing a report

The first session in a project writes .marionette/.gitignore, so reports stay out of version control without setup. Committing one, for example to attach it to a pull request, is a deliberate choice: copy the folder out, or remove that file.

The CLI writes the same records. Each command is its own process, so each one gets its own folder; --session only names it.

From a run to a Patrol test

A run that passed is a test script already: steps.md lists every action with the key it targeted, and report.md says what was checked. Give both to your agent in a project with Patrol, and ask for a test that replays the run. Marionette explores the app; Patrol keeps the result in CI.

The marionette-flutter-to-patrol skill, from Add the Agent Skills, teaches the agent the rest: which widgets need keys, how each step maps to Patrol, and which assertions the report implies.

prompt
Write a Patrol test from .marionette/sessions/<run>/steps.md and report.md:
the same steps with the same keys, and the checks from the report as
expectations.