When the Diff Doesn't Fit, Decide What You're Willing to Not Review

Context

Mewtwo is an Elixir service that reviews pull requests with LLM agents. A webhook arrives, an Oban job fetches the PR's unified diff, and several agents — bugs, security, performance, architecture, readability — analyse it in parallel and report findings.

The whole design rests on one uncomfortable fact: a pull request has no size limit, and a model's context window does. Everything else in the pipeline is downstream of how that mismatch gets resolved.

Problem

A large PR is rarely large because someone wrote a lot of code. It is large because it contains a lockfile, or a vendored dependency, or a directory of generated assets. A 400KB diff might be 380KB of pnpm-lock.yaml.

So there are two questions, and they are not the same question:

1. How do you make the diff smaller?
2. What do you do when it is still too big?

The first is a compression problem with a clean answer. The second is a product decision disguised as an engineering one, and getting it wrong produces something worse than an error: a confident review of code the model never saw.

What was tried

The compression side went in first, as a staged pipeline in Mewtwo.Compression, each stage narrowing the diff before the next one runs.

Line-level compression keeps changed lines and three lines of context around each, merging overlapping windows so context isn't duplicated:

needed_indices =
  changed_indices
  |> Enum.flat_map(fn idx ->
    Range.new(max(0, idx - 3), min(length(lines) - 1, idx + 3))
  end)
  |> Enum.uniq()
  |> Enum.sort()

File summarization collapses any unchanged run longer than 50 lines into a marker line — // ... 123 unchanged lines ... — so the model can see that a gap exists without paying for its contents.

Pattern grouping was supposed to be the third stage: detect a change repeated across many files (the same import added in twenty places, the same rename) and represent it once with examples. It shipped as a module that walked the diff, normalized each changed line into a pattern by masking string literals and numbers, counted occurrences, filtered to patterns appearing more than three times — and then did this:

defp apply_grouping_to_diff(diff, _grouped_patterns) do
  # For now, return diff as-is
  # Pattern grouping requires tracking which lines match patterns and replacing them
  # This is complex due to maintaining diff structure
  # TODO: Implement full pattern replacement with examples
  diff
end

Eighty lines of analysis feeding a function that returns its input.

Token accounting throughout uses a deliberately dumb estimator: byte_size/4, rounded up. It started out with separate heuristics for prose and code and a type hint parameter to select between them.

What failed

Even with the pipeline in place, a lockfile-heavy PR sailed straight through it. Removing unchanged context from a package-lock.json diff barely helps — nearly every line in it is a changed line. The compressed output was still enormous, and the request to the model was rejected outright, which is the least useful possible outcome: the full cost of fetching and compressing, and no review.

The pattern grouper failed differently. It was never wrong; it just never did anything. It cost a full pass over the diff and a page of code to produce an identity function, and it appeared in the pipeline's stages_applied metadata, so the logs claimed a stage had run.

The token counter's type hint failed in the most boring way possible: both branches converged on the same 4-bytes-per-token heuristic, so the parameter selected between two identical behaviours. Every call site passed :code out of politeness.

Final solution

The pattern grouper was deleted. Not finished, not stubbed with a warning — removed. The same pass took out a file_contents map that was threaded from the worker through compress/3 into the file summarizer, which had always ignored it as _file_contents. The stage list dropped from four entries to three. A stage that does nothing is worse than an absent stage, because it advertises coverage that doesn't exist.

The token counter shrank from 180 lines to 33, and the type hint stayed in the signature with the documentation stating plainly that it is ignored. Deleting a parameter every caller passes buys nothing; being honest about what it does costs one sentence.

In place of pattern grouping came the stage that actually solves the size problem: a truncator that drops whole files, cheapest first, until the diff fits. Files are sorted into tiers by reviewability:

# Lower tier is dropped first.
@generated 0
@asset 1
@snapshot 2
@test 3
@source 4

Lockfiles by name, anything under dist/, build/, vendor/, node_modules/, _build/, .next/, target/, coverage/, minified output and source maps are generated. Binaries and fonts and images are assets. Snapshots are their own tier. Tests rank above all of those but below hand-written source, which is dropped only when nothing else is left.

Within a tier, the largest file goes first, so each drop buys back as much budget as possible:

|> Enum.sort_by(fn {section, index} -> {section.tier, -section.tokens, index} end)

The important part isn't the ranking. It's what gets prepended to the surviving diff:

"""
# NOTE: this diff was truncated to fit the review token budget.
# #{length(dropped)} file(s) were omitted entirely and you cannot see their contents:
#{listed}#{extra}
"""

where each listed line names the file, its token count and the tier that condemned it — generated or vendored, binary or asset, snapshot, test, or source, over budget. The model is told, in the prompt, exactly which files it cannot see and why. Without that marker an agent reviewing a truncated diff will summarize the change as smaller than it is — and it will be right about the text it was given and wrong about the pull request.

Three budgets now guard three different boundaries. PRContext rejects a diff over two million tokens outright, on the grounds that fetching and compressing something that large is wasted work. diff_token_budget (100K) caps the compressed diff and drives truncation. max_prompt_tokens (180K) is a pre-flight ceiling checked in the agent spawner before any model call:

if prompt_tokens > max_prompt_tokens() do
  # Sending it anyway buys one 400 per agent and no review.

And one more guard, at the other end of the range — because compression can be too successful:

if String.trim(compressed) == "" do
  # Compressing to nothing would send the agents an empty diff and produce
  # a confidently empty review.

Technical reasoning

Dropping whole files rather than trimming every file evenly is the central bet, and it is a bet about what reviews are for. Trimming uniformly gives the model a shallow view of everything; dropping by tier gives it a complete view of the code a human would actually comment on. A reviewer who skipped the lockfile did not do a worse review. A reviewer who read 30% of every file did.

The reason the truncation marker matters more than the ranking is that an LLM has no way to know what was withheld from it. Every other component in this pipeline can detect its own degradation — a failed agent returns an error, a rate limit returns a reset time. A truncated prompt looks exactly like a complete prompt from the inside. So the omission has to be stated in-band, in the text, where the model will read it.

The pre-flight token check follows the same logic in reverse. The compressed diff is only one input; context, tool findings and instructions all add tokens on top of it, and only the spawner knows the assembled total. Checking there costs microseconds and converts a wasted API call per agent into a specific, actionable error.

Lessons learned

Delete the stage that doesn't work. A no-op pipeline stage is a lie in your metadata and dead weight in your logs. If it can't be finished now, its absence is more honest than its presence.

A parameter that selects between identical behaviours is not flexibility. Either the two paths differ or there is one path.

Compression and truncation are different problems. Compression is lossless-ish and always worth doing. Truncation is lossy and requires a policy about what you are willing to not review — that policy is a real design decision and deserves to be written down as tiers, not buried in a take/2.

Tell the model what you hid from it. Silent truncation doesn't produce a smaller review; it produces a confident review of a different pull request.

Estimate tokens crudely, then leave headroom. bytes/4 is not accurate. It doesn't need to be — the budgets sit far enough below the real ceiling that an estimate within ten percent is fine, and a simple heuristic that everyone understands beats a precise one nobody trusts.