from cc_transcript import parse
type(parse(b""))cc_transcript.models.TranscriptThe Rust engine
Why cc-transcript executes on a single compiled engine, and how hand-owned literals and golden fixtures pin its correctness without a second implementation.
Every hot path in cc-transcript executes in one place: _native, the pyo3 extension compiled into the wheel. It parses transcripts and bash command lines, runs the session-activity probe, extracts command prefixes, interprets every declarative spec — FilterSpec, ScoreSpec, MiningSpec, and the sentiment lexicon — and hosts the compiled CLI behind the cc-transcript console script. The extension is a hard requirement, not a fast path: every parsing and spec surface imports it directly, with no fallback implementation and no switch to route around it. The package root itself stays lazy — importing cc_transcript.ids or cc_transcript.tools pulls the standard library only — so the extension loads on first touch of a parsing or spec surface, not at import time.
Every parse returns a native Transcript view. There is no other parser.
Why the second implementation is gone
Earlier releases shipped four of these pieces twice: a Rust fast path plus a pure-Python reference behind a Backend protocol, held at verified parity, with portability gates (is_portable, the CC_TRANSCRIPT_DISABLE_RUST environment variable) deciding which interpreter ran a given spec. That design earned its keep when there was real divergence to guard against. The sentiment lexicon originally ran spaCy on the Python side and UDPipe in Rust — two NLP toolchains that could disagree token for token — and a parity-asserted reference was the only honest way to ship that.
v12 removed the divergence at its source. The surface-only lexicon replaced both toolchains with a deterministic shared tokenizer over two vendored TSVs, leaving nothing that could disagree. What remained of the dual build was pure cost: every semantic change landed twice, every release re-pinned a parity suite, and the portability gates split the spec language into a portable subset and a Python-only remainder that users had to reason about. So the Python interpreters are deleted. One engine, one behavior, one source of truth.
How a spec crosses the boundary
A FilterSpec is filters as data — an ordered tuple of Clause rules. spec_to_json serializes it to the JSON contract the engine compiles (compile_spec in rust/crates/core/src/filter.rs) and evaluates per line during parsing, so events that fail the spec are dropped before a Python object is ever built. Pass a drop spec to parse or stream and the rejected events never cross the FFI boundary.
from cc_transcript import build_spec, keep_only, drop_junk, drop_short
from cc_transcript.filterspec import spec_to_json
spec = build_spec(keep_only("user", "assistant"), drop_junk("structural"), drop_short(2))
print(spec_to_json(spec)[:120]){"clauses":[{"predicate":{"kind":"KindIs","kinds":["assistant","user"]},"action":"drop","applies_to":[],"negate":true,"lScoreSpec and MiningSpec follow the same shape. score_spec_to_json feeds the score_short_circuit and score_post_process entry points that bracket model inference; mining_spec_to_json feeds mine, which runs the Rust detectors over a session’s parsed events.
The post-parse moment crosses the same boundary. keep, labels_for, apply_spec, and annotate_spec match a spec against events you already hold — materialized views in a list — by handing each event’s borrowed native view to a per-event matcher, no serialization involved. That is what cc-transcript show and grep do: parse once, then filter the same events repeatedly. Same spec, two moments, one interpreter — which also means one regex dialect: a spec regex compiles as Rust regex at both moments, so foo$ misses "foo\n", and lookaround or \Z raises ValueError at match time instead of compiling.
The lexicon needs no spec at all: rust/crates/py/src/lexicon.rs compiles the two packaged TSVs in at build time, and tokenization runs a real NLP pipeline — tokenizer, POS tagger, lemmatizer, and dependency parse — over the embedded UD-English-EWT model, via udpipe-rs. The model ships in the wheel (cc_transcript/sentiment/data/en-ewt.udpipe) and compiles into the extension the same way the TSVs do: no download, no runtime file read. One engine is what makes a real pipeline affordable again — there is no Python toolchain left to hold at parity. The tokenizer splits contractions the way Universal Dependencies does (can't yields ca and n't), and Lexicon.has_hit is negation-aware: a negated token’s polarity is sign-flipped, so “isn’t great” reaches the negative floor, not the positive one. Polarity lookup itself stays keyed on the token’s surface — never its lemma — and the whole thing remains a straight FFI call. For the typed view, cc_transcript.nlp.analyze returns Token objects carrying the form, lemma, POS tag, codepoint span, polarity, and negation flag.
Wheels without a toolchain
The extension targets abi3-py313, so one wheel per platform covers CPython 3.13 and everything after it. Prebuilt wheels ship for macOS (arm64 and x86_64) and Linux (x86_64 and aarch64); there are no Windows wheels. uv add cc-transcript pulls a binary wheel, and a Rust toolchain enters the picture only when you build from source, where maturin compiles the crate.
Where Python still runs
One Python execution point survives the pivot, and it is not a fallback.
CallableReviewFormat. Mining’s plugin point: an arbitrary regex plus a Python extractor, for review-comment shapes no declarative format covers. The engine invokes both from inside its single pass through a pyo3 callback — single-threaded, under the held GIL — and propagates any Python exception. The declarative RegexReviewFormat executes entirely in Rust.
Model inference stays Python-side too, by design: FilteredEngine runs every deterministic score stage in Rust and hands only the undecided buckets to your InferenceEngine.
Warning
Rust’s regex crate has no lookaround, and a RegexReviewFormat whose pattern uses it raises at mine time. Lookaround belongs in a CallableReviewFormat, whose pattern is ordinary Python re.
How correctness is pinned
Cross-backend parity used to be the correctness mechanism: run both implementations, assert identical output. With one implementation, the suite pins behavior against hand-owned sources and frozen fixtures instead. uv run pytest runs all of it.
Hand-owned literals, one source. The shared constants — the CC-protocol markers, the mining ids, separators, and confidence floors, the command tables, and the corrections DDL — are hand-owned in Rust, under rust/crates/core/src/literals/. The crate compiles them in, and embedded_literals() exposes the whole table to Python as one dict; cc_transcript/literals.py binds each Python-side constant through a typed accessor (literal_str, literal_float) rather than keeping its own copy of the value. There is no generator — scripts/build_rust_literals.py is gone, and the generation direction inverted: to change a literal, edit the Rust source and rebuild. tests/test_literals_parity.py is the drift gate: every Python mirror must equal its embedded_literals() entry, the manifest must be fully covered (a native literal with no Python mirror fails), and no cc_transcript module may re-declare a distinctive literal as its own source string.
Vendored TSVs. The lexicon data lives once, as package data: afinn-en-165.tsv (provenance in its header) and domain_overrides.tsv under cc_transcript/sentiment/data/. The crate include_str!s the same files Python reads, so editing a TSV changes both sides at the next build. tests/test_lexicon_parity.py holds the files to their format and pins the tokenizer against a fixture of contraction splits and Unicode edge cases.
One embedded grammar. tree-sitter-bash compiles into the extension and nowhere else: command.py re-exports the native Command, CommandLine, and Occurrence views, and tree-sitter left the runtime dependencies with the flip. One deliberate per-surface contract survives it — Command.matches and has_arg keep exact Python re semantics (lookaround works), while the CLI grep and every spec regex run the Rust dialect.
Golden regression fixtures. tests/testdata/filter_golden.json, score_golden.json, and mining_golden.json freeze the engine’s output over hand-built batteries: presets and regex edge cases for the filter, the full four-stage spec for the score executor, one case per detector (plus near-misses, case folds, and unicode) for mining. command_golden.json and splice_golden.json do the same for the bash layer: batteries of parsed command lines and byte-range splices frozen at the switch. The goldens were captured from the historical Python reference at the moment of the switch — proven equal then — and the suite asserts the engine still reproduces them, so a behavior change fails loudly with no second implementation to compare against.
That is the trade. The reference implementation is gone, and in its place the correctness story is data — hand-owned literals, vendored tables, and frozen output — which drifts louder and costs one implementation instead of two.