Many AI tools for UI tests put a model in control of each test run. Thus, each run has a model cost. Also, the model can go through the app in a different sequence each time.

Optics Framework uses the model at a different point. Your AI agent goes through the app one time through the Optics MCP server. Then it writes a CSV test suite. After that, each run replays the suite and uses no model.

I did this procedure on saucedemo.com, a sample web site for tests. The agent was Claude Code with claude-opus-5-5, and the server was Optics Framework 1.11.0. This post shows the recorded session, the files that the agent wrote and the data from 10 replays.

Watch the recording on optics-framework.org

The recorded agent session at 1.5× speed. The left panel shows each Optics tool call, and the right panel shows the browser that the agent controls.

The prompt

I gave Claude Code one prompt and one MCP server. Skills and user settings were off. Thus, the agent had only the information from Optics.

Use the optics tools to open https://www.saucedemo.com in Playwright with headless false.
Sign in as standard_user / secret_sauce, add the Sauce Labs Backpack to the cart,
open the cart, check out as Ava Birch with postcode 560001, finish the order,
and check that "Thank you for your order!" appears.
Then write it as an Optics suite in ./checkout, run optics dry_run on it,
and tell me how to run it.

What the MCP server gives the agent

optics mcp is a Model Context Protocol server. In version 1.11.0 it has 41 tools. 3 tools control the session, and 38 tools are Optics keywords such as press_element, enter_text and assert_presence. The agent can also read these MCP resources:

• optics://session/{id}/elements: a short list of the elements on the screen. Each item has a label, the permitted actions and the position on the screen.

• optics://session/{id}/screenshot and optics://session/{id}/source: the screen as an image, and the full UI tree.

• optics://project-format: the rules for the suite files.

• optics://keywords: the full list of keywords.

On a web page, an item in the element list can also have a loc value. Optics makes this CSS selector in the browser. Then Optics makes sure that the selector finds only that element. This is the item for the Login button on saucedemo.com:

{"i": 3, "label": "Login", "cls": "input", "bounds": [494, 302, 786, 351], "act": ["tap"], "rid": "login-button", "loc": "[data-test=\"login-button\"]"}

The agent used a loc value for each of its 11 clicks and text entries.

The server has no tool to record or save a test. The agent writes the files, and it uses the rules from optics://project-format. The server also does not show the tools that tap a screen coordinate. On a different screen size, a coordinate from a screenshot can point to a different element.

The session

The time for the session was 71 seconds, with 32 model turns. The agent made 21 Optics tool calls and read the project format one time. It read the element list 6 times, one time on each new page. Claude Code showed a model cost of $0.44.

First, the agent did the full checkout in the browser. It found "Thank you for your order!" on the last page. Then it wrote 4 files and ran optics dry_run. The dry run passed.

The files

The test case uses 5 modules in sequence. The modules file has one row for each action, and all the values are variables:

checkout/modules/modules.csv:

module_name,module_step,param_1,param_2
Open Site,Launch App,${site_url}
Login,Enter Text,${username_field},${username}
Login,Enter Text,${password_field},${password}
Login,Press Element,${login_button}
Add Backpack To Cart,Press Element,${add_backpack_button}
Add Backpack To Cart,Press Element,${cart_link}
Add Backpack To Cart,Press Element,${checkout_button}
Enter Checkout Details,Enter Text,${first_name_field},${first_name}
Enter Checkout Details,Enter Text,${last_name_field},${last_name}
Enter Checkout Details,Enter Text,${postal_code_field},${postal_code}
Enter Checkout Details,Press Element,${continue_button}
Finish Order,Press Element,${finish_button}
Finish Order,Assert Presence,${order_complete_header}

The elements file holds the test data and the locators. For most elements, the agent put the verified loc value first and the label second:

checkout/test_data/elements.csv (part):

Element_Name,Element_ID,Element_ID_2
site_url,https://www.saucedemo.com,
username,standard_user,
first_name,Ava,
username_field,"[data-test=""username""]",Username
login_button,"[data-test=""login-button""]",Login
checkout_button,"[data-test=""checkout""]",Checkout
finish_button,"[data-test=""finish""]",Finish
order_complete_header,text=Thank you for your order!,"[data-test=""complete-header""]"

For a different user or a different product, write new values in this file. The modules stay the same.

These 4 files are the full result of the session. You can read them, review them in a pull request and keep them in Git with the code of your app.

The replay

Then I ran the suite 10 times with optics execute. The environment had no API keys. All 10 runs passed. The shortest run was 2.8 seconds, the longest run was 4.2 seconds, and the median was 3.3 seconds. Each time includes the start of Python and the start of the browser.

Watch the recording on optics-framework.org

3 recorded replays of the suite at 1× speed, with the terminal output of each run.

To count the model calls, I recorded each DNS lookup and each network connection from the Optics Python process. In all 10 runs, the record was empty. Only the browser connected to the network, to load saucedemo.com. To make sure that the record works, I sent one request to example.com from the same process. The record showed that request.

The source code agrees with this result. Optics makes an LLM client only when you set ai_self_heal to true. The value is false if you do not set it.

Data from the agent session and from the 10 replays.

When the page changes

A suite from an agent has the same problem as a suite from a person: the app can change. To simulate a refactor, I replaced the login button locator in elements.csv with [data-test="login-btn-v2"]. That selector is not on the page.

The run passed. Optics did not find the first locator. Then it tried the second locator, the label "Login". The log shows each locator that Optics tried:

Locating element: [data-test="login-btn-v2"] of type: CSS...
Locating element: Login of type: Text...
Pressing element 'Login'

The time for this step was 10.2 seconds. With a correct first locator, the time for the same step is approximately 0.1 seconds. Optics waits for the first locator until its time limit, and then it tries the label. The run passes with the second locator, but the run is slower. When the log shows a fallback, update the first locator.

Do it yourself

Python 3.12 (minimum) and an MCP client are necessary. These steps use Claude Code.

1. Install Optics Framework with the MCP server: pip install "optics-framework[mcp]".

2. Install the browser engine: optics setup --install playwright.

3. Add the server to Claude Code: claude mcp add optics -- optics mcp.

4. Give the agent a prompt with the same parts as the prompt above: the URL, the steps, the last check and the folder for the suite.

5. Read the files that the agent writes. Examine them as you examine a pull request.

6. Run the suite: optics execute checkout.

For Cursor, VS Code and Claude Desktop, refer to Let your AI agent build tests. For CI, set headless: true in config.yaml and refer to Run tests in CI.

Limits

• The quality of the suite is a function of the model and the prompt. Read the files before you commit them.

• optics dry_run examines the files only. It does not open the app. Run optics execute one time before you use a new suite in CI.

• loc values are available on web pages only. On Android and iOS, the agent uses the ids and labels from the element list.

• Loops, conditions and data files are not MCP tools, because they operate only in a full suite run. The agent can write them into the modules.

• The HTTP transport (optics mcp --transport http) has no authentication. Keep it on 127.0.0.1 or on a network that you control.

For an introduction to Optics Framework, read Introducing Optics Framework. To find error screens that no step examines, read Your tests passed. Your app didn't.

• Docs: Let your AI agent build tests and the MCP server reference

• Source: github.com/mozarkai/optics-framework

A checkout test signs in, opens a pair of trail runners, adds them to the cart and checks that “Added to cart” appears. Every step passes. Then the app shows “Session expired”, because last night’s build broke the token refresh. The run ends green and nobody looks at the last screen.

Every step passes and CI goes green. The last screen says the shopper was signed out.

A test checks for the things you expect to see. Error screens turn up wherever they like: Android’s “Birch has stopped” dialog, an “isn’t responding” prompt, a page that says “Something went wrong”, a “No internet connection” banner. You could assert against each one after every step, and your suite would double in size and still miss the next message someone adds.

Optics, the open-source test automation framework, has a feature for this called error definitions. You list the messages that mean something went wrong, and Optics looks for them on the screen at the end of every run.

The error definitions file

Add a CSV anywhere in your Optics project:

error_code,match_string,description,severity
ERR_CRASH,has stopped,The app crashed,critical
ERR_CRASH_AGAIN,keeps stopping,The app crashed again,critical
ERR_ANR,isn't responding,The app froze,critical
ERR_SESSION,Session expired,The user was signed out,high
ERR_SERVER,Something went wrong,Server error page,high
ERR_NETWORK,No internet connection,Offline,medium

match_string is the text to look for, and error_code is the short name you'll see in reports. description and severity are optional. Optics copies them into the report so whoever reads a failure knows what it means.

Optics recognises the file by its error_code and match_string columns, and optics execute uses it on every run. You don't switch anything on or call a keyword.

What happens at the end of a run

After the last test case, Optics reads the final screen, finds “Session expired” and records the match. A CI step turns it into a failed build.

After the last test case, Optics captures the screen once more and saves a screenshot and the page source with the run’s output. It then looks for every match_string in the screen's text. Matching ignores case and finds the text anywhere, so Session expired also matches "Your session expired. Please sign in again."

For each match, Optics logs a warning, writes the details to execution_output/detected_errors_.json and adds a failing test case named on-screen-error-detection to the JUnit report. The warning looks like this:

ON-SCREEN ERROR DETECTED — code: ERR_SESSION | matched_on: match_string | match_string: 'Session expired' | severity: high | The user was signed out

I ran a checkout test against a small web demo shop. The terminal said Passed: 1 | Failed: 0, and this file appeared next to the report:

{
  "timestamp": "2026-10-16T06:05:38.634183+05:30",
  "detected": [
    {
      "error_code": "ERR_SESSION",
      "matched_on": "match_string",
      "match_string": "Session expired",
      "description": "The user was signed out",
      "severity": "high",
      "detected_at": "end_of_run"
    }
  ]
}

Which text Optics reads

Optics keeps only the text a person can see, then checks it against every match_string.

The check reads the UI tree the driver already provides, with no OCR, and keeps the text a person can see. On Android and iOS that is the text, content-desc, label, value, hint and name attributes. On the web Optics strips every tag and keeps the page's text. It skips ids, class names and bounds, so a button with the id error_retry or a div with class="error" can't set off a match. The whole check costs one page-source call at the end of the run.

Optics matches the error_code column too. If your app prints its own codes, such as "Error E1042", use E1042 as the code and Optics will catch it even when the wording around it changes.

Make CI fail on it

Optics adds each match to the JUnit report as a failing test case named on-screen-error-detection, so any CI tool that reads JUnit reports lists it as a failed test, with the error code and the message. To fail the job itself, add one step after the suite that fails when the JSON file exists:

- name: Run the suite
  run: optics execute birch-tests
- name: Fail on error screens
  run: |
    if compgen -G "birch-tests/execution_output/detected_errors_*.json" > /dev/null; then
      cat birch-tests/execution_output/detected_errors_*.json
      exit 1
    fi

The suite step passes. The next step fails and prints which message was on screen.

Optics writes the file only when something matched, so a clean run passes the step. When the build goes red, the job log already shows the error code and the message, and you don’t have to open a screenshot to find out why. Run it on a fresh checkout, since an old execution_output folder still holds the previous run's file.

Why it helps

You get the check on every test you already have, and none of them change. One file covers a project’s Android, iOS, web and TV runs, because Optics reads each driver’s UI tree the same way. Anyone can keep the list up to date: when support starts seeing a new error in tickets, they add a row to a CSV and it goes through review like any other change. And when a run fails, the report says “Session expired”, which is what the user saw.

Most teams do this by hand. In Playwright you write an afterEach hook that reads the page text. With Appium or Espresso you write a teardown that inspects the screen, or a script that searches logcat for FATAL EXCEPTION. Logcat catches a crash, but an app showing "Session expired" looks healthy to the operating system. Firebase Test Lab's Robo crawler reports crash dialogs it meets while it explores an Android app, which is useful, though it runs apart from your regression journeys. In Optics the list is a data file next to your tests, and it runs on every platform your suite runs on.

Writing good match strings

• Pick a phrase that only an error would show. “has stopped” works. “Error” matches too much, such as a help page about error messages.

• Copy the exact words from a screenshot of the error and keep them short. Matching is plain text, so * and .* match only themselves.

• List both Android crash messages. Android says “Birch has stopped” the first time an app crashes and “Birch keeps stopping” when it crashes again.

• Leave out text your app shows on purpose. A match anywhere on the last screen counts.

Tips

• Leave the app open at the end of the run, so Optics has a screen to read. If your Suite Teardown closes the app, move that step out of teardown.

• Optics checks the screen where the run ends. For a message that can come and go in the middle of a flow, assert the screen you expect next.

• In Python or Robot Framework, call Add Error Definitions once, then Capture And Detect Errors anywhere you want a check, including halfway through a test.

Try it

curl -fsSL https://optics-framework.org/install | sh
optics quickstart

Add error_definitions.csv to the project and run optics execute. The docs are at optics-framework.org/docs/run/error-detection and the source is at github.com/mozarkai/optics-framework. Later posts in this series cover the locator fallbacks, AI agents over MCP, and testing phones that can’t run automation software.

A designer changes “Sign in” to “Log in”, or a developer renames the button’s id in a refactor, and the nightly run fails even though the app works. Someone then spends the next morning updating locators.

At Atlassian, flaky tests caused up to 21% of master build failures in the Jira frontend repo. Mobile teams often write the same flow separately for Android, iOS and the web. Many newer AI testing tools put a model in charge of every step of every run, so each run costs money and can take a different path through the app.

Optics, licensed under Apache 2.0, runs on your laptop or CI runner. It has no paid edition or account, and it sends no telemetry.

Four ways to create a test. Each one writes the same plain-text suite.

The test format

A test row describes one action a person takes in the app, and Optics works out how to perform it on the current screen.

module_name,module_step,param_1,param_2
Sign In,Enter Text,${email_field},ava@birch.shop
Sign In,Press Element,${sign_in}
Sign In,Assert Presence,Picked for you
Add To Cart,Press Element,Add to cart

These rows form modules, which are reusable sequences of steps. A test case lists the modules to run in order, and an elements file maps names like sign_in to the locators that find them. All three are plain text in your repository, so product managers can read them and changes go through pull requests like any other code.

Four ways to create a test

All four write the same files, and one project can mix them:

• Write it in a text editor or a spreadsheet.

• Record it in optics live, an interactive session next to your device. Each step runs on the device as you type it, and /save stores the steps that worked as a module.

• Describe it in plain English. Press Ctrl+N in optics live, type add trail runners to the cart, and an AI model drives the app one keyword at a time.

• Delegate it to your AI agent. Add optics mcp to Claude Code, Cursor or another MCP client, and every keyword becomes a tool the agent can call while it explores the app and writes the suite files.

optics live runs each step on the device as you type it, and /save writes the steps into the suite.

Teams that prefer code can use the same keywords from Python or Robot Framework.

How Optics finds an element

You can give an element several locators. Optics tries them in order, cheapest first:

1. XPath, through the app’s UI tree.

2. The visible text, in the same tree.

3. OCR, which reads the text from a screenshot.

4. Image matching against a small picture of the element.

Element_Name,Element_ID
sign_in,//*[@resource-id='login']
sign_in,Sign in

With these two rows, a refactor that renames the button’s id breaks the XPath, and Optics falls back to the button’s text. Text painted into a banner image never reaches the UI tree, so OCR is the only way to read it. The report for each step names the method that found the element, which tells you when a locator has gone stale.

After a redesign the XPath misses and Optics finds the button by its text. OCR reads a product name from a banner image.

AI in Optics

Optics uses AI to write tests. A model or an agent explores the app and leaves behind keyword rows for you to review, and optics execute replays those rows without calling a model. The browser-use team made a similar choice when they built a deterministic follow-up to their AI browser agent, because in their words "pure LLM agents were slow, expensive, and unpredictable."

AI self-heal is the one exception. You can switch it on as a last step for when every locator fails, and the log records each action the model takes.

Optics calls a model only while you write a test.

Supported platforms

A project’s config.yaml names the target, and the same test files run on:

• Android and iOS, through Appium, on emulators, simulators and real phones.

• Websites, through Playwright or Selenium.

• Android TV, Samsung Tizen and LG webOS, through Appium, using remote-control keys.

• Locked production phones, through a Bluetooth mouse and keyboard and a camera pointed at the screen. Nothing gets installed on the phone, so a journey tested in CI can also monitor the live app.

Teams that already run Appium, Playwright or Selenium keep their drivers and devices, since Optics drives them directly.

The same test on a phone, a TV and a browser.

Running a suite

optics execute prints a live tree of every test case, module and step. This is a real run on an Android emulator:

A real optics execute run on an Android emulator.

A run leaves behind a JUnit report for CI, screenshots with a box around every element it found, and an exit code of 1 when a test fails. You can list error messages such as Session expired in a definitions file, and Optics looks for them on the last screen to catch failures no step asserted.

optics dry_run checks a suite without a device, and loops, conditions and data files run one flow with many inputs. To connect Optics to other tools, Invoke API calls your backend mid-test, optics serve exposes the keywords over REST, and optics generate converts a suite to pytest or Robot Framework code.

Try it

Optics needs Python 3.12 or newer and runs on macOS, Linux and Windows:

curl -fsSL https://optics-framework.org/install | sh
optics quickstart
optics execute my-optics-project

On Windows, install with irm https://optics-framework.org/install.ps1 | iex. pip install optics-framework works on any system. The quickstart asks whether you're testing a mobile app or a website, installs the engine it needs, and creates a project you can run.

• Docs: optics-framework.org/docs

• Source: github.com/mozarkai/optics-framework

Later posts in this series cover the locator fallbacks, AI agents over MCP, and testing phones that can’t run automation software. Send bug reports and questions to the GitHub issues.