Zacian: Building an AI-Powered Code Review Agent with Multi-Stage Verification

Introduction

Code reviews are essential for maintaining software quality, but they're often a bottleneck in the development workflow. They require human reviewers to have deep domain knowledge, catch security vulnerabilities, identify performance issues, and ensure architectural consistency — all while managing time constraints.

Zacian is an automated code review agent that addresses this challenge. It's an Elixir/Phoenix application that integrates with GitHub to automatically review pull requests using Claude AI across multiple specialized domains, providing comprehensive findings without the manual review overhead.

In this post, I'll walk you through Zacian's architecture, how it works, and the engineering decisions behind it.

What is Zacian?

Zacian is a GitHub-integrated code review bot that:

Triggers automatically when a PR is labeled with a configured review label
Gathers context by fetching the diff, full file contents, and recent PR history
Runs multi-stage reviews through Claude across specialized passes (bugs, security, quality, performance, architecture)
Verifies findings to reduce false positives through specialist agents
Consolidates results using a judge stage to merge duplicates and order by severity
Posts results as a single comprehensive PR comment with markdown rendering

The bot is built as a single Elixir/Phoenix application deployed on AWS ECS, handling concurrent reviews with proper isolation and graceful degradation.

Architecture Overview

High-Level Pipeline

GitHub Webhook (PR labeled)
  ↓
ReviewSupervisor (DynamicSupervisor - concurrency capped)
  ↓
ReviewJob (GenServer per PR - handles retries)
  ├→ Diff Agent (fetch & validate PR diff)
  ├→ Context Agent (fetch full file contents)
  ├→ History Agent (fetch recent merged PRs on same files)
  ↓
Reviewer (1 Claude call → raw findings with categories)
  ↓
Specialists (5 concurrent agents - one per category)
  ├→ Bug Specialist
  ├→ Security Specialist
  ├→ Quality Specialist
  ├→ Performance Specialist
  └→ Architecture Specialist
  ↓
Judge (merge duplicates, drop weak findings, order by severity)
  ↓
GitHub.post_comment (post final findings on PR)
  ↓
RunHistory (persist to Postgres for analytics)

Key Design Principles

1. Isolation: Each PR review runs as an isolated supervised process. One PR's failure can't affect another.

2. Concurrency Bounded: A DynamicSupervisor caps concurrent reviews (default 5) to protect Claude/GitHub rate limits.

3. Graceful Degradation: Context/History failures degrade review quality but don't fail the review. Only Diff failures trigger retries.

4. Verification + Judgment: Raw findings pass through specialist verification and a judge stage to reduce false positives.

Core Modules

`Zacian.ReviewJob` (GenServer)

The heart of the system. Each PR gets its own ReviewJob GenServer that:

Orchestrates the entire review pipeline
Implements retry logic (up to 3 attempts with backoff)
Manages the state transitions of each stage
Publishes stage completion events to PubSub for live UI updates

The state machine flows through: pending → fetching_context → reviewing → verifying → consolidating → posting → complete.

`Zacian.Claude`

A shared Bedrock InvokeModel client used by Reviewer, Specialists, and Judge. Key features:

Wraps AWS Bedrock SDK for Claude model invocation
Extracts token usage from Bedrock's response metadata
Passes token usage to RunTracker for cost accounting
Handles configurable model selection (persisted in Settings)
Supports model ID prefixing (bare claude-opus-5 → us.anthropic.claude-opus-5)

`Zacian.Reviewer` & `Zacian.Agents.Specialist`

Reviewer makes a single Claude call with the full PR context (diff + file contents + history) asking for structured findings tagged by category (bug, security, quality, performance, architecture).

Specialists (5 concurrent agents) independently verify findings in their category:

Each specialist re-examines only findings tagged for its category
May drop false positives or sharpen summaries
Never invents new findings
Results are merged by the Judge

`Zacian.Judge`

Consolidates findings post-verification:

Merges findings describing the same root cause
Drops findings that failed verification
Orders results by severity (critical → high → medium → low)
Produces the final list posted to GitHub

`Zacian.RunTracker` & `Zacian.ReviewStore`

RunTracker is a GenServer that:

Accumulates per-stage token usage and status
Publishes stage completion events via PubSub
Derives run verdict (complete/degraded/exhausted)
Calculates wall time and finding counts

ReviewStore holds in-memory review results (capped at 20) for the UI. It subscribes to RunTracker's PubSub topic.

`Zacian.RunHistory`

Persists runs and token usage to Postgres:

Subscribes to RunTracker PubSub topic
Writes runs + per-stage usage/cost asynchronously (non-blocking to review pipeline)
Enables analytics and spend tracking across restarts
Uses raw Postgrex (no ORM) for minimal overhead

`Zacian.Pricing`

Prices token counts using operator-configured rates:

Rates are configuration, never guessed
Missing rates yield token counts but no cost
Supports per-million-token pricing for input/output tokens separately

Data Flow: A Real Example

Let's trace through a PR labeled for review:

1. Webhook arrives: GitHub sends pull_request.labeled event → POST /webhooks/github

2. Deduplication: ReviewDedupe checks if {repo, pr, head_sha} is already in-flight (prevents duplicate processing from webhook redeliveries)

3. Job starts: ReviewSupervisor.start_review/2 spawns a new ReviewJob GenServer

4. Context gathering (parallel):

Diff Agent: Fetches PR diff from GitHub API, validates size (≤400KB)
Context Agent: Fetches full contents of up to 20 changed files (≤400KB total, best-effort)
History Agent: Fetches up to 5 recently merged PRs touching the same files (best-effort)

5. Reviewer call: Single Claude call with structured context:

You are an expert code reviewer. Review this GitHub PR:

[diff + file contents + recent history]

Provide structured findings in JSON:
{
  "findings": [
    {
      "file": "src/user.rs",
      "line": 42,
      "category": "bug" | "security" | "quality" | "performance" | "architecture",
      "severity": "critical" | "high" | "medium" | "low",
      "summary": "...",
      "fix": "..."
    }
  ]
}

6. Specialist verification (parallel): 5 agents independently verify their category:

Bug Specialist: Re-examine "bug" category findings
Security Specialist: Verify "security" findings
Quality Specialist: Check "quality" findings
Performance Specialist: Validate "performance" findings
Architecture Specialist: Review "architecture" findings

Each specialist can drop or refine findings, but not invent new ones.

7. Judge consolidation:

Merge findings describing the same issue
Drop any findings that failed verification
Order by severity (critical → high → medium → low)

8. GitHub comment: Post final findings as a single PR comment with formatted markdown

9. Persist: RunHistory subscribes to RunTracker's PubSub and writes to Postgres asynchronously (never blocks the review path)

Tech Stack

| Component | Technology | |-----------|------------| | Framework | Phoenix 1.8 + LiveView | | Language | Elixir 1.17 | | Server | Bandit HTTP server | | Database | PostgreSQL + raw Postgrex (no ORM) | | AI | AWS Bedrock + Claude | | GitHub API | Req HTTP client | | Concurrent Tasks | Task.Supervisor, DynamicSupervisor, GenServer | | State | PubSub, in-memory ReviewStore, PostgreSQL | | Frontend | LiveView + Tailwind CSS + HeroIcons |

Why Elixir?

Concurrency: GenServer + DynamicSupervisor provide natural isolation per PR
Reliability: Supervisor trees guarantee graceful degradation
Hot reload: LiveView enables real-time UI updates as stages complete
Fault tolerance: Isolated processes prevent cascading failures
Low latency: Concurrency without overhead

Configuration & Deployment

Required Environment Variables

GITHUB_TOKEN              # Bearer auth for GitHub API
GITHUB_WEBHOOK_SECRET     # HMAC-SHA256 verification
BEDROCK_TOKEN             # Bearer auth for AWS Bedrock
CLAUDE_MODEL              # Model ID (default: claude-opus-5)
BEDROCK_REGION            # AWS region (default: us-east-1)
BEDROCK_GEO               # Geo routing: us/eu/au/global (default: us)
MAX_CONCURRENT_REVIEWS    # Concurrency cap (default: 5)
SUPABASE_DB_URL           # Postgres URI (optional; in-memory fallback)
MODEL_RATES               # JSON pricing config (optional)
PORT                      # HTTP port (default: 4000)
SECRET_KEY_BASE           # Phoenix session signing (required in prod)

Example Model Rates Configuration

{
  "claude-opus-5": { "input": 12.0, "output": 60.0 },
  "claude-sonnet-5": { "input": 3.0, "output": 15.0 }
}

Deployment

Runs on AWS ECS (shared app platform):

Image: ECR your-apps:zacian-latest
Cluster: dt-apps
Host: zacian.example.com
Deployment: Push to master → GitHub Actions builds & force-deploys ECS service

The Review Console (LiveView)

Zacian includes a rich web UI for browsing reviews and monitoring:

Home (`/`)

Pull-request list with all stored reviews
Filterable by review type (bug/security/quality/performance/architecture)
Searchable by repo/PR number
Per-review confidence badge and run outcome

Reviewer View (`/reviews/:id`)

Review confidence (High/Medium/Low based on stage completeness)
Findings that need human review
Findings grouped by category
Blast radius estimation
Low-severity tail

Dev View (`/reviews/:id/dev`)

Same findings as a fix queue
Structured for developers to iterate and apply fixes

Comment Preview (`/reviews/:id/comment`)

Exact markdown that Zacian posted on the PR
What the PR author sees

Live Trace (`/trace` & `/trace/:run_id`)

Pipeline stages in real-time as they complete
Per-stage wall time and token usage
Manual rerun for exhausted runs
Model picker to switch Claude model for subsequent reviews

Dashboard (`/dashboard`)

Throughput: reviews per hour/day
Speed: median review time
Token usage: total input/output tokens
Spend: total cost and per-model breakdown
Data source: persisted Postgres history (or in-memory if no DB)

Key Features & Engineering Details

Idempotency

Problem: GitHub can redeliver webhooks; users can re-label PRs. Without idempotency, we'd post duplicate comments.

Solution: ReviewDedupe tracks {repo, pr, head_sha} → active ReviewJob, preventing duplicates. Findings are stored regardless of post success, so even if posting fails, findings aren't lost.

Retry Logic

Context/History failures: Degrade gracefully (less context for Reviewer)
Diff failures: Trigger full pipeline retry (up to 3 attempts with backoff)
Specialist/Judge failures: Treated like degradation

Token Accounting & Cost Tracking

Every Claude call records:

Input tokens (from Bedrock usage block)
Output tokens (from Bedrock usage block)
Cache hit/write tokens (currently always zero; ready for prompt caching)

Token counts are attributed to the pipeline stage that made the call:

Reviewer stage
Each Specialist stage
Judge stage

Costs are derived from operator-configured per-model rates. Missing rates don't block operation — they just yield "N tokens, cost unknown."

Size Guards

Diff: ≤400KB (reject if larger)
File contents: ≤400KB total across all touched files
PR history: Up to 5 recent PRs (best-effort)

These guards prevent overwhelming Claude with too much context.

Graceful Degradation

| Stage | Failure Behavior | |-------|-----------------| | Diff fetch | Retry entire pipeline (up to 3x) | | Context fetch | Skip, continue with diff only | | History fetch | Skip, continue with diff + context | | Reviewer call | Fail pipeline | | Single specialist | Drop findings from that category | | Judge | Fail pipeline | | GitHub post | Log failure; findings still in ReviewStore |

Manual Trigger & CLI

For ad hoc reviews without a webhook:

mix zacian.review owner/repo 123

Runs the review pipeline synchronously. Skips context/history agents (diff-only) to avoid long-running CLI commands.

Observability & Debugging

Live UI (/trace): Watch pipeline stages complete in real-time
Review history: Browse past reviews, inspect findings, check tokens per stage
Postgres history: Durable run records survive restarts
PubSub events: Stage completion publishes to RunTracker topic; UI subscribes for live updates

Production Learnings

What We've Learned

Specialist verification is critical. Raw Claude findings have ~30% false positives; specialists reduce this to <5%.
Token accounting is essential. Without per-stage tracking, you can't optimize prompt sizes or identify cost outliers.
Graceful degradation wins. Missing context is better than no review at all. A degraded review is still useful.
Postgres persistence matters. Keeping only in-memory history is fine for testing, but production needs durable analytics.
Concurrency bounds prevent thundering herds. 5 concurrent reviews is enough to be useful without rate-limit issues.

Future Enhancements

Prompt caching: Cache PR context across multiple specialist calls to reduce tokens
Budgets & alerting: Enforce spend ceilings per org/day
Custom rules: Let orgs configure review policies
Code checkout: For expensive checks, optionally boot PR code
Finding persistence: Store findings in Postgres, not just runs

Conclusion

Zacian demonstrates how to build a production-grade AI agent on Elixir/Phoenix:

Isolation: Supervised processes prevent cascading failures
Concurrency: DynamicSupervisor + GenServer handle parallel work cleanly
Degradation: Graceful fallbacks for missing context or failed stages
Observability: Token accounting, PubSub events, persistent history
Verification: Multi-stage pipeline reduces false positives

Building it on Elixir meant less boilerplate around concurrency and state management, letting us focus on the review logic itself.

If you're considering AI agents for your workflow, Zacian's architecture is a good blueprint: isolate work, bound concurrency, degrade gracefully, and invest in verification.

Questions or feedback? Feel free to reach out on Twitter or via email — I'd love to hear how you're using AI for code quality.

Zacian - Building an AI-Powered Code Review Agent with Multi-Stage Verification