Reviews
What a review does once it starts: the deep review that looks for bugs, how far it reads beyond the diff, static analysis, the coding agent instructions it reads as conventions, and the walkthrough comment. Shopify themes get more checks, and what you can do with a finding once it is posted is on Findings and fixes.
Deep review
The deep review looks for bugs, not style. Several passes with a frontier model look for problems (how many, and how far they read, depends on the depth), and a verifier model checks each finding before it is posted. Turn it off for the team or one repository with Deep bug review; the rules pass then runs alone. A pull request over 3,000 changed lines (the default limit) skips the deep review, and the check run says so. When a change touches Shopify code, the passes and the verifier also get built-in Shopify knowledge. Pick the find, verify and description models on the Auditor page; the choices are limited to models your team Directives allow. If a Directive later blocks a chosen model, the review uses an allowed one and the check run says so.
Review depth
| Depth | What runs |
|---|---|
| Standard | Three passes over the diff and the changed files, read in different orders. Findings at least two passes agree on are kept (a High finding needs only one), and a verifier checks each one. Cheapest. |
| Thorough | Default. Four specialized passes (logic and edge cases, security, data and concurrency, API contracts) that can read, search and list the whole repository at the pull request's head, so they follow callers and definitions outside the diff. Static analysis runs first and its results are handed to the passes, and a verifier checks every finding. Costs more per review. |
With Incremental review on, a review reads only the commits since the last one, except after a rebase or force-push, when 300 or more files changed, or when @auditor review finds no new commits: those read the whole pull request.
Every request either depth makes is billed to the Auditor key and listed in Logs, so the cost of a review is visible per run. Each repository can override the depth.
Static analysis
On Thorough reviews, unless Static analysis is off on the Auditor page, Auditor runs the repository's own type checker and linter (for example tsc --noEmit and ESLint, when the repository is set up for them) plus Semgrep's default rules, and reports only new errors on lines the pull request changed. Findings name the tool, such as tsc TS2345 or eslint no-unused-vars.
When the repository is a Shopify theme, Theme Check runs too, with the same Shopify CLI the theme previews use: shopify theme check --output json on the theme's folder, with the theme's own .theme-check.yml if it has one. A repository counts as a theme only when it holds layout/theme.liquid and config/settings_schema.json (at its root or in one folder, such as theme/) and the pull request changes the theme's files; any other repository never installs or runs it. Only offenses on lines the pull request added are reported, as theme-check MissingTemplate and the like: errors are High, warnings Medium, info Low. One exception follows from Shopify sending a page only the {% stylesheet %} CSS of the files it renders: when the pull request moves or deletes a class's rule and another, untouched file still uses the class out of the rule's reach (ValidScopedCSSClass), the offense is reported on the line that moved the rule, or in the review's summary when the rule was only deleted. A shopify.theme.toml is set aside while it runs, so a default environment in it cannot stop the check.
The checks run in an isolated sandbox: a separate microVM that holds no credentials. The pull request's files are copied in, not cloned with a token; the network reaches package registries only; and dependency install scripts are turned off. Sandbox compute is platform-funded for now; model requests stay billed to the Auditor key.
Coding agent instructions
The deep review reads what the repository already tells coding agents: AGENTS.md, CLAUDE.md and .github/copilot-instructions.md at the root, and the Markdown files directly in .cursor/rules/ (up to 20, with a rule's globs kept as what it applies to). They are conventions for context, so the review doesn't report as a bug what they ask for; they are not checked like .rules/, which the rules pass enforces. Like .rules/, they are read from the base branch, so a pull request can't change the instructions it is reviewed with. Each file is trimmed to 4,000 characters and all of them to 10,000; lines that only import another file (such as @AGENTS.md) are left out, so a CLAUDE.md that just points at AGENTS.md adds nothing. The walkthrough comment ends with the files the review read. To review without them, turn off Coding agent instructions under Deep review on the Auditor page's Reviews tab. It is a team setting: it applies to every repository, and a repository can't override it.
Re-reviews
Every review after a pull request's first checks what earlier reviews found: a finding whose flagged code is gone, or whose suggested change was applied, is resolved and its thread closed. Findings still open are not posted again. What else a re-review posts is the Re-reviews setting under When to review, which a repository can override:
| Setting | A re-review posts |
|---|---|
| Post every new finding | Default. Whatever it finds that isn't already open on the pull request. |
| Verify, and post only new High | New High findings only. New Medium and Low ones are counted in the check run and not posted, so a fix-and-push loop doesn't raise new nits each round. @auditor review full posts everything. |
A re-review's new findings are marked Previously missed when they sit on code that hasn't changed since the review before (more than three lines from any change since): the earlier review could have found them. They are listed apart in the review and in the walkthrough comment's review state. After a rebase or force-push, when what changed can't be told, nothing is marked.
Walkthrough comments
With Walkthrough comment on, Auditor posts one comment on each reviewed pull request: a short summary, a file-by-file table of the changes and, when a change spans several components, a sequence diagram. Later reviews update the same comment instead of adding new ones. Under the toggle, choose which of those parts to include, and turn off Update on later reviews to write it once, on the first review only. With no part checked, no walkthrough is posted.
From the second review on, the walkthrough also carries the pull request's review state, rewritten after every review whether or not it posted anything: what is open (new, previously missed, and still open from earlier reviews), what was resolved since the review before, and how many Medium and Low findings a verify-only re-review didn't post. Editing the comment in place means no new notification.