LLM cost control

Gate an LLM verdict behind cheap deterministic filters so the model fires only when an edit already looks suspicious — instead of burning tokens on every turn.

An LLM hook that fires on every edit burns tokens fast. You want a model to catch a weakened test — a real assertion turned into assertTrue(True), a no-op mock, a skip to make a failing test pass — without paying for a model call on edits that could never be one. This example spends a call only when it earns its keep.

"""Gate an LLM verdict behind cheap deterministic filters to catch weakened tests cheaply."""

from __future__ import annotations

from captain_hook import (
    Agent,
    Content,
    Event,
    SourceEdits,
    TestFile,
    llm_gate,
)

# Keep LLM hooks cheap. Gate the model behind cheap deterministic filters, run the
# small model, fire at most once per session, and skip planning subagents. Most
# turns never reach the model, so they never spend a token.
llm_gate(
    "Does this edit weaken a test, turning a real assertion into assertTrue(True), a "
    "no-op mock, or a skip, to make a failing test pass? Block only if unambiguous.",
    message="This looks like a weakened test: {reasoning}",
    # Cheap pre-filters first: only edits to a test file that actually touch
    # assert/mock/skip ever reach the model.
    only_if=[SourceEdits(lang="py", include_tests=True), TestFile(), Content(r"\b(assert|mock|skip)\b")],
    skip_if=[Agent("Explore|Plan|general-purpose")],
    events=Event.PostToolUse,
    model="small",
    max_fires=1,
)

Verified by replay

The model verdict carries no inline test — an LLM judgment needs a recorded fixture or a live session, so you exercise it by replaying real sessions under uvx capt-hook test rather than asserting it deterministically here.

What it does

Four levers bound the cost:

only_if=[SourceEdits(lang="py", include_tests=True), TestFile(), Content(r"\b(assert|mock|skip)\b")]  # cheap pre-filters: only test-file edits touching assert/mock/skip reach the model
skip_if=[Agent("Explore|Plan|general-purpose")]  # skip planning subagents
model="small"  # pick the cheaper model
max_fires=1  # stop after the first intervention

Deterministic conditions run first, so the model only sees edits that already look like a weakened test. Most turns never reach the model, so they never spend a token.

Run it yourself

uvx capt-hook --hooks docs/examples test

See also