Findings and fixes

Replying to a finding

Reply in the thread under any Auditor finding and Auditor answers there, usually within a minute; 👀 on your reply means it's working on it. It reads the finding, the code at the pull request's head, the rule the finding came from and the thread so far.

• Ask why, or what to change, and it explains from the code.
• Push back and it weighs your reasoning against the code. If you're right it says so, dismisses the finding, resolves the thread and adds what it learned to the repository's learnings. If not, it explains why the finding stands.
• Say it's intentional, or that you won't fix it here, and the finding is dismissed: it is never raised again on that pull request. An intentional pattern also becomes a learning.
• Ask to leave it open and later reviews won't resolve the thread, even after the code changes.

Anyone can ask a question, but only people with write access to the repository can dismiss a finding or keep it open this way. Auditor answers the latest reply in a thread, up to four times per thread, and each answer is one model call billed to the team like a review. Replies are a team setting: turn them off for every repository with Answer replies in review threads under When to review on the Auditor page's Reviews tab. Commands such as @auditor review work in pull request comments, not in review threads.

Fixing in your coding agent

Each finding's comment ends with a collapsed Fix in your agent block. Open it for a prompt written for a coding agent: the file and lines, what is wrong, any suggested replacement, and to change only what the finding needs, in a commit of its own. In the review comment's table, and in Auditor's reply to @auditor review, each open finding has a Fix link, and the block starts with the same link. Prompts cover one finding each, so every fix starts in a fresh agent session and lands as its own small commit.

Fix opens the finding's page on Inference Bureau: the finding as its comment says it, the whole prompt, and a card for each way to fix it. Cursor opens the prompt in Cursor, which shows it to you before it runs. Claude Code and Codex give one command to run in a checkout of the pull request's branch, such as claude "$(curl -fsSL https://app.inferencebureau.com/f/<token>/prompt)", which fetches the prompt and starts the agent with it. Auditor fixes that one finding as @auditor fix would and posts the result in its thread; that needs a member of the team, signed in. For any other agent, copy the prompt from the page or from the comment's code block.

The page opens without signing in for anyone who can see the review: its address holds a random key made for that finding, and it shows the finding and its prompt and nothing else of the repository. It is never indexed. To keep findings off the web entirely, or to link straight to Cursor as before, change Fix links in reviews under Fixes on the Reviews tab: Cursor links each finding to Cursor (its links hold at most 8,000 characters, so a longer prompt leaves the explanation out and points the agent at the review comment), and No Fix links leaves only the prompt to copy. Under a white label, Fix links go to Cursor and the prompts leave the MCP server out.

Fixes with @auditor fix

Comment @auditor fix on a pull request and Auditor fixes its open findings in two parts. First, the suggested changes posted with findings are applied, wherever the flagged code is still as it was flagged. No model is called for these, so they cost nothing.

Then, when Write fixes for findings without a suggested change is on (under Fixes on the Reviews tab; on by default), Auditor fixes up to 8 open findings that have no suggestion. It works like a careful engineer with read, search and edit access to the pull request's code at its latest commit: it checks each finding is real, makes the smallest change that fixes it, and keeps callers consistent. It skips a finding it judges wrong, or one that needs a decision from you, and says why. It never edits .github/, lockfiles or generated files, and changes at most 12 files. Findings about the store's content are not fixed in code.

With Check fixes before committing on, the fixed code goes to an isolated sandbox, where the repository's type checker and linter, Semgrep and, for a Shopify theme, Theme Check run on the changed files. Errors on lines Auditor changed go back to it for one more round. A fix they still flag opens as a draft pull request, with the errors listed.

So a fix has at most two rounds, and each has limits. A round gets 150 seconds and 30 turns of tool calls; when either runs out, Auditor stops editing and reports on what it got to, and a finding it didn't finish is skipped. A round stopped by a budget, a spend cap or the balance may have left a fix half made, so its edits are dropped and its findings are reported as not fixed; if that happens in the second round, the first round's edits are kept. The reply says when a fix stopped early, and why.

Auditor then replies on the pull request with what it fixed, what it didn't and why, what checked it, and what it cost. By default the fixes are a pull request into your branch, so nothing changes until you merge it; Where fixes go can commit them straight onto your branch instead. Branches from forks are never changed. If someone pushes while a fix is being made, nothing is committed.

Writing fixes uses the deep review's find model and is billed to the Auditor key like a review. Its requests are in Logs as session = athf_<repository id>_<pull request>, it counts toward the spend caps, and it shows as Fixes in a repository's cost.

Tests with @auditor tests

Comment @auditor tests on a pull request and Auditor writes unit tests for what it changes. Add what to focus on after the command, such as @auditor tests the retry path. It first finds how the repository tests (its framework, where tests live, how they are named and how they mock), then tests the behaviour the pull request adds or changes: the main path, the edge cases and the errors its diff handles. It only creates and edits test files, at most 8, so the code under test is never changed; when the code looks wrong, it leaves that case out and says so.

The tests then run in the same isolated sandbox fixes are checked in, with dependencies installed and install scripts off: with Vitest or Jest, found from each test file's nearest package.json, or pytest for Python. Set Write tests on the Reviews tab, or tests.command in the settings file, to run them some other way (the test files are added to the end of the command). When they fail, Auditor reads the output and fixes the tests, up to three rounds in all.

Passing tests go where fixes go: a pull request into your branch, or a commit onto it when Where fixes go says so. Tests that still fail, or that couldn't be run, always come as a draft pull request with the output, never onto your branch. It is billed like a fix, as session = athf_<repository id>_<pull request>, and counts toward the spend caps. Turn Write tests off and the command isn't answered.

Resolving merge conflicts

When a pull request conflicts with its base branch, comment @auditor resolve conflicts and Auditor merges the base branch in. A file only one side changed takes that side; a file both changed is merged line by line, and only the lines that still conflict go to the model, with the pull request's description and the base branch's commit messages, so it can keep both changes: a renamed function, a moved argument, a new field. Each file's resolution is explained in the reply.

Nothing half merged is ever committed. Auditor stops, and says which files need a person, when one side deleted a file the other changed, when a binary file or a submodule changed on both sides, when the base branch changed a GitHub Actions workflow (GitHub doesn't let Auditor commit those; use Update branch), when more than 15 files conflict, when it can't settle a conflict without a decision only you can make, or when the result still holds conflict markers or JSON that no longer parses. Auditor merges the base branch as it is at that moment, and the merge is a real merge commit, committed straight onto your branch as GitHub's own Resolve conflicts does, so GitHub sees the base branch as merged and the conflict is gone. If someone pushes to the branch while it works, nothing is committed. A pull request with no conflicts gets a pointer to Update branch instead. It is billed like a fix.

Findings over MCP

With the inference-bureau MCP server connected, a coding agent reads findings itself, for any repository Auditor reviews, not only Shopify themes. list_findings lists the findings open on a pull request, given its number or the branch the agent is on; get_finding gives one in full with a prompt for fixing it; get_review gives the latest review; and resolve_finding closes a finding once it is dealt with, resolving its thread on GitHub (all the tools).

Ask the agent to “fix the open Auditor findings on this branch” and it can list them, fix each, and resolve them as it goes. A finding resolved as fixed counts as acted on for tuning, and if the flagged code is still there after the push, the next review raises it again. One resolved as wont_fix, with a note saying why, is dismissed, and Auditor learns nothing from it.

Stale comments

On every review Auditor tidies up after itself. A finding whose flagged code is gone from the file has its thread resolved and its comment struck through with Resolved in the commit that fixed it. A rule finding whose rule no longer applies (removed from .rules/, no longer in a team ruleset for the repository, switched to a dry run, or now excepted for that path) is resolved and marked No longer applies with the reason. Once a ticket is found, the “No ticket key found” comment is collapsed as outdated. Threads Auditor resolves itself never count as dismissals.