Judge

judge.VerdictLike

The structural shape a judge’s verdict must expose to be persisted.

Usage

judge.VerdictLike()

Read-only by design, so both plain attributes and property aliases satisfy it — an app whose model names the bits differently (cc-steer’s is_steering/what_claude_did) adapts with two properties.

Attributes

category: str: The single best-fitting category label.
summary: str: One neutral sentence summarizing the judged action.
confidence: float: The judge’s probability that its accept-vs-reject call is right.
rationale: str: One short clause explaining the call.
accepted: bool: Whether the verdict accepts the row.
canonical_key: str | None: The canonical, normalized key of the durable rule the verdict names, or None when the verdict names no durable rule.

judge.run_verdicts()

Runs judge over every row’s prompt and persists each verdict as it lands.

Usage

Source

judge.run_verdicts(rows, prompt_for, judge, persist, *, concurrency)

Incremental by construction: a row whose judge call raises JudgeError is counted as failed and left unpersisted, so the next pass retries it; every other row’s verdict persists as soon as its call completes. Any other exception is a programming error and propagates, cancelling the pass. Generic over the verdict payload, so the same fan-out serves triage- and refinement-shaped passes.

Parameters

rows: Sequence[Mapping[str, object]]: The rows to judge.
prompt_for: Callable[[Mapping[str, object]], Awaitable[str]]: Builds one row’s prompt; async, so prompts may hydrate their context window first.
judge: Callable[[str], Awaitable[V]]: Turns one prompt into a verdict payload, e.g. structured_judge(...).
persist: Callable[[Mapping[str, object], V], Awaitable[None]]: Persists one row’s verdict, e.g. store.record_verdict applied.
concurrency: int: The maximum number of concurrent judge calls.

Returns

tuple[int, int]: The pass’s (judged, failed) counts.

judge.sample_audit()

Draws the deterministic stratified audit sample over judged rows.

Usage

Source

judge.sample_audit(
    judged_rows,
    *,
    accepts,
    rejects,
    seed,
    quotas,
    remainder_kind,
    oversample_share=0.3
)

The draw is seeded and pure, so an evaluator can reproduce the exact core set by calling it with the same inputs. Per side (accepted/rejected): every kind in quotas gets its quota (None means exhaustive), the remainder budget goes to remainder_kind, and within each subsampled kind oversample_share of the draw oversamples the judge’s lowest-confidence verdicts.

Parameters

judged_rows: Sequence[Mapping[str, object]]: Rows in ~cc_transcript.mining.store.FeedbackStore.judged() shape for one pass.
accepts: int: The audit budget for accepted rows.
rejects: int: The audit budget for rejected rows.
seed: int: The iteration’s deterministic sampling seed.
quotas: Mapping[str, int | None]: Per-source-kind audit quotas; None audits the kind exhaustively.
remainder_kind: str: The source kind the leftover budget draws from.
oversample_share: float = 0.3: The fraction of each subsampled draw spent on the judge’s lowest-confidence verdicts.

Returns

AuditSample: The sampled rows, split into the uniform core and the oversample.

judge.Metrics

The full mechanical evaluation of one prompt version.

Usage

Source

judge.Metrics(
    prompt_version,
    total,
    judged,
    accepted,
    golden,
    core_accepts,
    core_rejects,
    pool_accepts,
    pool_rejects,
    by_kind,
    disagreements
)

Attributes

prompt_version: int: The judge prompt version evaluated.
total: int: The corpus row count.
judged: int: How many rows carry a judge verdict at this version.
accepted: int: How many of those are accepted.
golden: GoldenResult: The golden-set gate outcome.
core_accepts: AuditEstimate: Audited precision numerator/denominator over the uniform core.
core_rejects: AuditEstimate: Audited contamination numerator/denominator over the uniform core.
pool_accepts: AuditEstimate: The same estimate over every audited accept (cumulative pool).
pool_rejects: AuditEstimate: The same estimate over every audited reject (cumulative pool).
by_kind: Mapping[str, tuple[int, int]]: (judged, accepted) counts per source kind, descriptive only.
disagreements: tuple[Disagreement, …]: Every audited row where auditor and judge disagree.

Attributes

Name	Description
contamination	Audited genuine-accept rate over the uniform core’s rejects.
contamination_upper	The exact one-sided 95% upper bound on contamination.
precision	Audited precision over the uniform core’s accepts.
recall_hat	The derived estimate of the fraction of genuine accepts accepted.

contamination

Audited genuine-accept rate over the uniform core’s rejects.

contamination: float | None

contamination_upper

The exact one-sided 95% upper bound on contamination.

contamination_upper: float | None

precision

Audited precision over the uniform core’s accepts.

precision: float | None

recall_hat

The derived estimate of the fraction of genuine accepts accepted.

recall_hat: float | None

judge.AuditEstimate

One binomial estimate from audited rows.

Usage

Source

judge.AuditEstimate(audited, hits)

Attributes

audited: int: How many rows of the population carry an auditor verdict.
hits: int: How many of those the auditor accepted.

Attributes

Name	Description
rate	`hits / audited`, or `None` when nothing is audited.

rate

hits / audited, or None when nothing is audited.

rate: float | None

judge.exact_upper_bound()

Returns the exact (Clopper-Pearson) one-sided upper confidence bound.

Usage

Source

judge.exact_upper_bound(hits, n, alpha=0.05)

The smallest rate p such that observing hits or fewer successes in n trials has probability at most alpha — the rule of three’s exact generalization.

Parameters

hits: int: The observed success count.
n: int: The trial count.
alpha: float = 0.05: The one-sided significance level.

Returns

float: The upper bound on the true rate.

judge.GoldenRow

One frozen, hand-labeled row of the golden regression set.

Usage

Source

judge.GoldenRow(dedup_key, source_kind, text, expected, note)

Attributes

dedup_key: str: The content-derived key joining the row to feedback_events.
source_kind: str: The detector that produced the row.
text: str: The verbatim message, kept for human review of the fixture.
expected: bool: The frozen accepted-vs-rejected label.
note: str: One clause recording why the label holds.

judge.golden_result()

Gates one pass’s verdicts against the frozen golden fixture.

Usage

Source

judge.golden_result(golden, corpus_keys, judge_by_key, sha256)

Parameters

golden: Sequence[GoldenRow]: The fixture’s rows.
corpus_keys: set[str]: Every stored event’s dedup key.
judge_by_key: Mapping[str, Mapping[str, object]]: The pass’s verdicts in ~cc_transcript.mining.store.FeedbackStore.judged() shape, keyed by dedup key.
sha256: str: The fixture file’s digest, carried into the result.

Returns

GoldenResult: The gate outcome with every mismatched or unjudged row.

Raises

LookupError: If any golden row is missing from the corpus (drift).

judge.flip_pairs()

Compares two verdict passes row by row.

Usage

Source

judge.flip_pairs(earlier, later)

Parameters

earlier: Sequence[Mapping[str, object]]: The earlier pass’s rows in ~cc_transcript.mining.store.FeedbackStore.judged() shape.
later: Sequence[Mapping[str, object]]: The later pass’s rows in the same shape.

Returns

FlipReport: The overlap size and every side-changing row, ordered by dedup key.

judge.structured_judge()

Returns a prompt-to-verdict callable that plugs into run_verdicts().

Usage

Source

judge.structured_judge(response_model, *, tier, timeout=LLM_TIMEOUT)

Each call runs one structured completion on the cached default backend via spawnllm.extract(), which validates the response into response_model. Provider and transport failures — a backend call error, no ready backend, a timeout, or a non-conforming response — surface from the returned callable as ~cc_transcript.judge.verdicts.JudgeError, the one exception run_verdicts() counts as failed and retries next pass.

Example

>>> judge = structured_judge(Verdict, tier="medium")
>>> await run_verdicts(rows, prompt_for, judge, persist, concurrency=8)

judge.resolved_model()

The concrete model name the active backend runs for an abstract tier.

Usage

Source

judge.resolved_model(tier)

The verdict store keys on this string; it reflects whichever backend spawnllm.select_backend() resolves, so a judged corpus stays coherent within one backend environment.