Your tests passed. Your app didn’t.

Catch crash dialogs, “Session expired” and other error screens that no step checked for, with one CSV file in Optics.

A checkout test signs in, opens a pair of trail runners, adds them to the cart and checks that “Added to cart” appears. Every step passes. Then the app shows “Session expired”, because last night’s build broke the token refresh. The run ends green and nobody looks at the last screen.

Every step passes and CI goes green. The last screen says the shopper was signed out.

A test checks for the things you expect to see. Error screens turn up wherever they like: Android’s “Birch has stopped” dialog, an “isn’t responding” prompt, a page that says “Something went wrong”, a “No internet connection” banner. You could assert against each one after every step, and your suite would double in size and still miss the next message someone adds.

Optics, the open-source test automation framework, has a feature for this called error definitions. You list the messages that mean something went wrong, and Optics looks for them on the screen at the end of every run.

The error definitions file

Add a CSV anywhere in your Optics project:

error_code,match_string,description,severity
ERR_CRASH,has stopped,The app crashed,critical
ERR_CRASH_AGAIN,keeps stopping,The app crashed again,critical
ERR_ANR,isn't responding,The app froze,critical
ERR_SESSION,Session expired,The user was signed out,high
ERR_SERVER,Something went wrong,Server error page,high
ERR_NETWORK,No internet connection,Offline,medium

match_string is the text to look for, and error_code is the short name you'll see in reports. description and severity are optional. Optics copies them into the report so whoever reads a failure knows what it means.

Optics recognises the file by its error_code and match_string columns, and optics execute uses it on every run. You don't switch anything on or call a keyword.

What happens at the end of a run

After the last test case, Optics reads the final screen, finds “Session expired” and records the match. A CI step turns it into a failed build.

After the last test case, Optics captures the screen once more and saves a screenshot and the page source with the run’s output. It then looks for every match_string in the screen's text. Matching ignores case and finds the text anywhere, so Session expired also matches "Your session expired. Please sign in again."

For each match, Optics logs a warning, writes the details to execution_output/detected_errors_.json and adds a failing test case named on-screen-error-detection to the JUnit report. The warning looks like this:

ON-SCREEN ERROR DETECTED — code: ERR_SESSION | matched_on: match_string | match_string: 'Session expired' | severity: high | The user was signed out

I ran a checkout test against a small web demo shop. The terminal said Passed: 1 | Failed: 0, and this file appeared next to the report:

{
  "timestamp": "2026-10-16T06:05:38.634183+05:30",
  "detected": [
    {
      "error_code": "ERR_SESSION",
      "matched_on": "match_string",
      "match_string": "Session expired",
      "description": "The user was signed out",
      "severity": "high",
      "detected_at": "end_of_run"
    }
  ]
}

Which text Optics reads

Optics keeps only the text a person can see, then checks it against every match_string.

The check reads the UI tree the driver already provides, with no OCR, and keeps the text a person can see. On Android and iOS that is the text, content-desc, label, value, hint and name attributes. On the web Optics strips every tag and keeps the page's text. It skips ids, class names and bounds, so a button with the id error_retry or a div with class="error" can't set off a match. The whole check costs one page-source call at the end of the run.

Optics matches the error_code column too. If your app prints its own codes, such as "Error E1042", use E1042 as the code and Optics will catch it even when the wording around it changes.

Make CI fail on it

Optics adds each match to the JUnit report as a failing test case named on-screen-error-detection, so any CI tool that reads JUnit reports lists it as a failed test, with the error code and the message. To fail the job itself, add one step after the suite that fails when the JSON file exists:

- name: Run the suite
  run: optics execute birch-tests
- name: Fail on error screens
  run: |
    if compgen -G "birch-tests/execution_output/detected_errors_*.json" > /dev/null; then
      cat birch-tests/execution_output/detected_errors_*.json
      exit 1
    fi

The suite step passes. The next step fails and prints which message was on screen.

Optics writes the file only when something matched, so a clean run passes the step. When the build goes red, the job log already shows the error code and the message, and you don’t have to open a screenshot to find out why. Run it on a fresh checkout, since an old execution_output folder still holds the previous run's file.

Why it helps

You get the check on every test you already have, and none of them change. One file covers a project’s Android, iOS, web and TV runs, because Optics reads each driver’s UI tree the same way. Anyone can keep the list up to date: when support starts seeing a new error in tickets, they add a row to a CSV and it goes through review like any other change. And when a run fails, the report says “Session expired”, which is what the user saw.

Most teams do this by hand. In Playwright you write an afterEach hook that reads the page text. With Appium or Espresso you write a teardown that inspects the screen, or a script that searches logcat for FATAL EXCEPTION. Logcat catches a crash, but an app showing "Session expired" looks healthy to the operating system. Firebase Test Lab's Robo crawler reports crash dialogs it meets while it explores an Android app, which is useful, though it runs apart from your regression journeys. In Optics the list is a data file next to your tests, and it runs on every platform your suite runs on.

Writing good match strings

• Pick a phrase that only an error would show. “has stopped” works. “Error” matches too much, such as a help page about error messages.

• Copy the exact words from a screenshot of the error and keep them short. Matching is plain text, so * and .* match only themselves.

• List both Android crash messages. Android says “Birch has stopped” the first time an app crashes and “Birch keeps stopping” when it crashes again.

• Leave out text your app shows on purpose. A match anywhere on the last screen counts.

Tips

• Leave the app open at the end of the run, so Optics has a screen to read. If your Suite Teardown closes the app, move that step out of teardown.

• Optics checks the screen where the run ends. For a message that can come and go in the middle of a flow, assert the screen you expect next.

• In Python or Robot Framework, call Add Error Definitions once, then Capture And Detect Errors anywhere you want a check, including halfway through a test.

Try it

curl -fsSL https://optics-framework.org/install | sh
optics quickstart

Add error_definitions.csv to the project and run optics execute. The docs are at optics-framework.org/docs/run/error-detection and the source is at github.com/mozarkai/optics-framework. Later posts in this series cover the locator fallbacks, AI agents over MCP, and testing phones that can’t run automation software.