Four Ways to Get Zero Findings From a Working Model

Context

Mewtwo reviews pull requests by spawning several LLM agents in parallel — one for bugs, one for security, one for performance, and so on — each of which gets a compressed diff and a prompt and is expected to return a JSON array of findings. The models are Claude, invoked through AWS Bedrock, from Elixir.

The interesting engineering is supposed to be in the review logic: which context to fetch, how to score confidence, how to deduplicate findings that two agents both spotted. Instead, a full day of the build went into the couple of hundred lines that sit between the prompt and the parsed finding.

Problem

Every failure at that boundary presents identically. An agent runs, returns no findings, and the pipeline reports success with zero results.

That output is also completely legitimate: an agent that finds nothing wrong is doing its job. So "zero findings" covers, at minimum: the model was never called, the model returned an error, the model returned prose instead of JSON, the JSON was fine but the parser rejected it, or the code genuinely has no problems. All of them look the same in the logs and none of them are distinguishable from success.

What was tried

The first version of the Bedrock client was the obvious translation of the API docs into Elixir:

url = "https://bedrock.us-east-1.amazonaws.com/model/#{model_id}/invoke"

body = %{
  anthropic_version: "bedrock-2023-06-01",
  max_tokens: 4096,
  ...
}

with credentials and model ID read straight out of the environment at call time, defaulting to anthropic.claude-3-5-sonnet-20241022-v2:0.

Every part of that snippet is wrong in a different way, and each one produces a different error that the caller flattens into the same string.

bedrock. is the control plane. Invocation lives on bedrock-runtime., and posting an inference request to the control-plane host returns UnknownOperationException — an error that says nothing about hostnames and reads like the endpoint doesn't exist.

bedrock-2023-06-01 is not the wire-format version Bedrock accepts. The correct string is bedrock-2023-05-31, and it is deliberately not the same string as the anthropic-version header on the first-party Anthropic API. Two similar-looking constants for two similar-looking APIs, and swapping them fails at the schema level rather than the transport level.

The model identifier is the subtlest of the three. A bare name like anthropic.claude-haiku-4-5, or an on-demand anthropic.* ID, is rejected by the invoke endpoint. It wants a full inference profile ID — the kind with a region prefix and a version suffix, us.anthropic.claude-opus-4-5-20251101-v1:0. And since that string goes into the URL path, it needs encoding rather than interpolation.

On the response side, parsing was a single Jason.decode/1 over the whole response body, filtering to maps with a binary file and an integer line.

What failed

Once the request actually reached the model, the parser threw away the answer.

Models routinely wrap a JSON array in a Markdown code fence, or introduce it with a sentence, or both. Jason.decode/1 on a string that begins with ```json fails, the parser returned [], and the agent reported no findings. The model had done the work. The three lines that read its answer discarded it, silently, and reported that discard as a clean review.

The is_integer(line) filter added a second silent drop: a model that emits "line": "42" had every finding rejected by a guard clause.

And a third problem was latent in the same function, one severity worse:

{:ok, finding} = AgentFinding.new(...)

A hard pattern match on validation output, inside an Enum.map running in a Task. One malformed finding raises a MatchError, which takes down that task — and because the spawner awaits all of them with Task.await_many/2, it takes down every other agent's work along with it. Five agents' worth of successful analysis discarded because one of them emitted a null file path.

Final solution

The client became explicit about all three constants, with comments recording why each value is what it is, because none of them are guessable from the outside:

# Anthropic's wire format version on Bedrock. Not the same string as the
# direct Anthropic API's `anthropic-version` header.
@anthropic_version "bedrock-2023-05-31"

# Invoke lives on the bedrock-runtime host; plain `bedrock.` is the
# control plane and answers with UnknownOperationException.
url =
  "https://bedrock-runtime.#{region}.amazonaws.com/model/#{URI.encode_www_form(model_id)}/invoke"

Auth is a Bedrock API key sent as a bearer token, so requests are plain HTTP with an Authorization header rather than SigV4 signing. Region, model ID and token resolve from environment first, then application config, then a documented default.

Parsing became a candidate list, tried in order:

defp json_candidates(response) do
  trimmed = String.trim(response)

  [trimmed] ++ fenced_blocks(trimmed) ++ balanced_spans(trimmed)
end

The raw response first, since a well-behaved response should not pay for the fallbacks. Then any fenced code blocks. Then, as a last resort, the first balanced [...] or {...} span in the text — a small character scanner that tracks brace depth and string state, so a bracket inside a JSON string literal doesn't end the span early:

cond do
  esc -> keep.(depth, in_str, false)
  in_str and c == ?\\ -> keep.(depth, in_str, true)
  c == ?" -> keep.(depth, not in_str, false)
  in_str -> keep.(depth, in_str, false)
  c == open -> keep.(depth + 1, in_str, false)
  c == close and depth == 1 -> acc |> Enum.reverse() |> List.to_string()
  ...

Arrays are tried before objects, so [{...}] is read as a list of findings rather than as the single object it happens to start with. Decoded output is normalized from three shapes — a bare list, a %{"findings" => [...]} wrapper, or a single finding object — and string line numbers are coerced rather than rejected.

Individual findings now fail individually:

# Returns nil rather than raising when a finding fails validation: a single
# bad entry must not take down the agent (and with it every other agent
# awaited by Task.await_many/2).

And the case that started all of this — silence that means failure versus silence that means success — is now distinguishable, because an unparseable non-empty response logs a warning with its size and first 200 bytes:

Logger.warning(
  "Agent #{agent_name}: no JSON findings found in #{byte_size(response)}-byte " <>
    "response: #{String.slice(response, 0..200)}"
)

Along the way the client started returning token usage as a third element — {:ok, text, usage} — read from the response's own usage object rather than estimated. That produced one more decision worth recording: per-token prices are not hardcoded anywhere. Bedrock is partner-operated and priced separately from the first-party API, so the rates are configuration, and when they aren't set the system reports token counts and says the dollar figure is unavailable:

:no_rates -> base <> ", cost unavailable (BEDROCK_*_USD_PER_MTOK not set)"

A wrong cost number is worse than a missing one, because a missing one gets configured and a wrong one gets quoted.

Technical reasoning

The unifying mistake in all four bugs is treating the model boundary as a function call. A function call has a type signature; this doesn't. The request has three magic strings that must match values written in documentation, and the response is free-form text that usually contains JSON. Both sides need defensive handling, and the defenses are not symmetrical.

For the request, the defense is being explicit and annotated. Every one of those constants failed with an error message that pointed somewhere else — a control-plane hostname reporting an unknown operation, a version mismatch surfacing as a schema complaint. The comments in the final code exist because the error messages will not lead the next person to the answer.

For the response, the defense is layered tolerance with visible fallbacks. Accepting a fenced block is not being sloppy about contracts; it is acknowledging that the model's output format is a strong tendency rather than a guarantee, and that throwing away a correct answer because it arrived inside a fence is a worse failure than parsing loosely.

The per-finding isolation is really a lesson about Elixir's process model rather than about LLMs. Task.async plus Task.await_many gives clean parallelism and shared fate — the tasks are linked to the caller, so one crash collapses the batch. Anywhere output crosses from an untrusted source into a Task, validation has to return values, not raise.

Lessons learned

When success and failure produce identical output, that's the first bug to fix. Distinguishing "no findings" from "couldn't read the response" was worth more than any individual fix, because it made the other three findable.

Comment the constants you cannot derive. A hostname, a wire version, an ID format — anything whose wrong value produces a misleading error deserves a sentence explaining why it is what it is.

Parse LLM output as text that contains JSON, not as JSON. Try the strict path first, then the fenced block, then a balanced span, and log when all of them miss.

Coerce what you can, reject what you must, and never raise. A malformed line number is not a reason to lose four other agents' findings.

Prefer "unavailable" to a plausible wrong number. Especially for cost, which is the one output people will repeat without checking.