Many-shot watcher

retrain.manyshot.build_exemplar_system()

The byte-constant many-shot system prompt: seeded demonstrations under budget_chars.

Usage

Source

retrain.manyshot.build_exemplar_system(
    rows, *, budget_chars=EXEMPLAR_BUDGET, seed=SEED
)

Walks rows in a seeded permutation, greedily keeping each demonstration whose body fits the remaining budget and skipping any that would overflow, then wraps them in the fixed instruction frame. Deterministic in (rows, budget_chars, seed) and byte-identical every call, so score_frame() can reuse one prefix across every batch and hit the provider’s prompt cache.

retrain.manyshot.score_frame()

Score the frozen frame with a cached many-shot prefix and store its P(NO_STEER).

Usage

Source

retrain.manyshot.score_frame(
    frame,
    exemplars,
    *,
    version,
    model,
    budget_chars=EXEMPLAR_BUDGET,
    batch_size=BATCH_SIZE,
    seed=SEED,
    eval_root=None,
    max_cache_miss_streak=MAX_CACHE_MISS_STREAK
)

Builds one byte-constant exemplar system prompt from exemplars and scores the frame in batch_size batches against it, so every batch after the first reads the prefix from the provider’s prompt cache instead of re-sending it. Each batch’s per-context steer probabilities become P(NO_STEER) = 1 - p; the whole frame’s probabilities and their fire AUC land through ~cc_steer.retrain.evalset.write_probs(), paired-comparable with any trained watcher on the same frame.

Parameters

frame: EvalFrame

The frozen eval frame to score.

exemplars: Sequence[WatcherRow]

The demonstration pool the system prefix draws from (watcher train rows).

version: str

The registry version label the stored probs are keyed under.

model: str

The frontier model to run, for example claude-opus-4-6.

budget_chars: int = EXEMPLAR_BUDGET


Character budget for the exemplar prefix.

batch_size: int = BATCH_SIZE


Frame rows scored per cached call.

seed: int = SEED


Seed for the exemplar shuffle.

eval_root: Path | None = None

Eval root override; defaults to ~/.cc-steer/eval.

max_cache_miss_streak: int = MAX_CACHE_MISS_STREAK


Consecutive post-warmup cache misses that abort the pass.

Returns

Path

The path the per-row probabilities were written to.

Raises

ManyShotError

When a batch reply is unparseable or the prompt cache stays cold for max_cache_miss_streak batches in a row.