The app¶
The app is built with Panel and panel-material-ui. It reads the same files as the CLI, so you can switch between them freely: a writeup edited in your editor shows up in the app within a few seconds, and a batch launched from the CLI appears on the Batches page.
The current page and theme are part of the URL (?page=labels&theme=dark), so reloads and shared links keep them. The header shows how fresh the local copies of GitHub data are; its menu lists issues, labels and pull requests separately and refreshes all three. The Outbox button next to it counts the changes waiting to be sent. If a background job fails, a notification appears wherever you are.
How review pages work¶
Close review, Labels and PR review share one pattern: a list next to a detail pane, a decision per item, and an automatic move to the next item. Decisions that change GitHub are only staged; they wait in the Outbox until you send them in one batch.
Overview¶
Headline numbers (open, triaged, untouched, verified, recommended for closing, agent spend) and charts: writeups by category (click a bar to browse that category), triage progress over time, recommendations by confidence, review outcomes per model and open issues by GitHub issue type, split by whether they have been triaged. Active claims and validation errors are listed below.
Chat¶
Ask questions about the backlog: "which confirmed bugs involve Tabulator?", "how many feature requests were opened before 2022?", "show me issues the reviewers disputed". The chat agent (built on Lumen) queries DuckDB tables of issues, writeups and stats, and has tools to pull up an issue's full writeup or GitHub thread. Configure the model as chat_model in Settings.
Issues¶
A searchable, filterable table of writeups next to the selected issue. Drag the divider to resize the two sides. For each issue you see the GitHub issue type and labels, the rendered writeup, the cached GitHub thread, the reproducers with their logs and screenshots (with a button to rerun them), and the agent transcript. The Edit tab is a markdown editor that validates before saving. The review bar lets you change category, confidence and recommendation, then Verify or Dispute with a note.
Close review¶
Issues recommended for closing, with the category, confidence, who triaged and reviewed them, the proposed comment (editable) and the reproduction evidence: what the writeup claims, what the reproducer logs actually say, the screenshot, scripts and logs, and linked pull requests. For each issue choose Close with this comment, Re-review (with an optional note) or Keep open.
Each writeup also carries a second opinion from the decision model, which judges the issue, the reproducer output and the facts about cited fixes (their description and first release) without seeing the agent's conclusions: the category it thinks fits, whether to close, whether the cited fix covers the request, and whether the evidence is sound. Confident disagreements and weak evidence show as warnings, and Order → Most doubtful first puts the weakest evidence on top. Treat it as a second opinion, not a ranking: the Calibration page shows how well it has matched your decisions so far.
Closes go to the Outbox. Re-reviews are agent jobs and start from Send re-reviews at the bottom. A re-review whose job failed shows as re-review failed with the error and a Retry button, instead of counting as sent. Issues with activity on GitHub since they were triaged carry a warning, and when the run contradicts the writeup a Send to re-review button appears next to the evidence.
Labels and PR review¶
See Label review and PR review.
Outbox¶
Everything staged for GitHub in one list: closes (with their comments), label and issue type changes, and PR comments. Items with a warning, such as new activity since triage or new commits since a PR was reviewed, start unchecked. Sending re-checks each issue on GitHub and holds a close if comments arrived since triage, unless you switch on Close even with new activity. Unstage selected sends items back to their review page.
The History tab lists every change ai-triager made on GitHub, newest first, with the exact commands. Select one to see what undo would run (reopen the issue and delete the comment, restore the previous labels and type, or delete a PR comment) and undo it after a confirmation. Undoing also puts the item back on its review page.
Calibration¶
How far each model's confidence held up against your decisions. Every decision you make on a suggestion (closing or keeping a recommendation, staging or dismissing labels, finding a PR ready or asking for changes) is appended to .triage/decisions.jsonl with what the model predicted at the time, so clearing or re-running reviews never loses it. Import past decisions adds history from before the log existed: closes on GitHub of issues whose writeup recommended closing, verdicts in replies/*.md, and earlier label and PR decisions.
- Close recommendations compares each triage model's self-reported confidence with how often its recommendations were accepted, and how often the writeup still needed corrections.
- Triage verdicts checks whether the decision model's second opinions (below) separate the close recommendations you accepted from those you rejected, and writeups that needed corrections from clean ones. AUC 0.5 is chance.
- Label and type suggestions plots predicted probability against how often you agreed, with the Brier score and expected calibration error.
- Per-label thresholds learns, for each label with enough decisions, the add and remove thresholds that would have matched you best (favouring precision). Switch on Use learned per-label thresholds to have label review use them.
- PR checks compares the scores of PRs you found ready with those you asked to change.
Batches¶
Preview which issues a batch would claim, launch triage or review jobs with a model, runner, size and parallelism, follow running jobs with a live feed of tool calls, and stop them.
Sync¶
Scheduled triage syncs on an interval, triages issues opened since scheduling started that have no writeup yet (up to a set number per run), and writes a digest of each run, which you can read here. It runs in its own process, so it keeps going when the app is closed. Below that: the issue list, writeups whose issue changed after triage, and the sync history.
Settings¶
Models for each task, API keys, skills, the execution sandbox and guard (with a Test sandbox button), spending limits, guard activity, the agent deny list and writable paths, external runner commands, and project maintenance actions. Saving checks every model name and refuses names that cannot work, such as openrouter:anthropic:claude-sonnet-5-5 (OpenRouter needs provider/model); Test models makes one tiny request per role. Other pages only offer models that pass this check. Guard activity lists every command the guard judged, with a filter for blocked ones. Model and agent settings are saved to settings.json; sandbox, budget and schedule settings to triage.toml.
