Autonomous issue triage that gathers its own evidence
The problem
A product produced a steady stream of errors in Sentry. Someone had to read each one, decide whether it was a real bug, a duplicate, or noise, reproduce it, file a ticket, and decide what happened next.
That work is repetitive, easy to drop, and easy to get wrong: two people file the same issue twice, or a system acts on the text of a report without seeing the app, so the fix targets the wrong thing.
What I built
An autonomous triage service. A scheduled job starts the run on one machine. The run is the coordinator: it reads new error groups, walks the case list, and decides exactly one action per case.
The action set is small and explicit. For a new problem it files one issue. For a repeat it links the case to the issue that already covers it, so one fault never becomes two tickets. For a case that needs code it starts a fix lane, a Codex build job that opens a draft pull request.
The evidence is visual: the run drives the staging build through Playwright and captures screenshots as it goes. They land in one thread per case with the analysis, and the operator works from them: approve, reject, or reply with an instruction such as mark this a duplicate, reference a related issue, or do something else. The reply is free text, so the operator is never boxed into yes or no.
How it works
- The scheduler starts the run on a fixed daily schedule.
- The run reads new error groups from Sentry, untriaged by default.
- It walks the case list, one case at a time, and decides one action per case.
- It opens the staging app with Playwright, drives the failing path, and captures screenshots.
- It writes one ledger row, one thread and one ticket, then posts the analysis and screenshots together.
- If the case needs code, the fix lane asks the model through a single local proxy that holds the keys and picks the provider, then opens a draft pull request.
- The operator approves, rejects or redirects in the thread, and that answer closes the case.
How I kept it reliable and safe
One action per case is the core rule, enforced by the ledger, so a restart never files anything twice.
All model calls go through one local door, so keys live in one place and the provider can change without touching the fix lane.
Every change lands as a draft pull request and waits for a person.
My role
I designed and built this system end to end: the coordinator, the ledger, the Sentry, chat and GitHub handling, the Playwright evidence step, the fix lane, and the model proxy.
What I'd measure next
Whether the chosen action is right the first time, and how many cases close without a human reopening them. Also how often the operator overrides the proposed action.