LLM cache

llmcache.LlmCacheMode

How CachingTransport treats each request against the store.

Usage

llmcache.LlmCacheMode()

llmcache.LlmCacheSettings

The [llmcache] section: default cache mode and key schema version.

Usage

Source

llmcache.LlmCacheSettings()

llmcache.CachingTransport

Record/replay layer keyed on sha256(schema_version, method, scheme, host, port, path, sorted-query, body).

Usage

Source

llmcache.CachingTransport(
    inner,
    mode,
    schema_version,
    db_path,
    lock=anyio.Lock(),
    stack=AsyncExitStack(),
    store=None
)

Headers are excluded from the key by design (auth and session noise never perturb replay), and only 2xx responses are recorded. Streaming responses are fully buffered: the body is read into memory before it is stored. The sqlite store lives under load(AthomeSettings).cache_root / "llmcache" / "llmcache.db" and opens lazily on the first request, closing when the transport does.

Parameter Attributes

inner: httpx.AsyncBaseTransport
mode: LlmCacheMode
schema_version: int
db_path: Path
lock: anyio.Lock = anyio.Lock()
stack: AsyncExitStack = AsyncExitStack()
store: Store | None = None

llmcache.RetryTransport

Retries 429/5xx/transport errors with capped exponential backoff (default 3 tries).

Usage

Source

llmcache.RetryTransport(inner)

Parameter Attributes

inner: httpx.AsyncBaseTransport

llmcache.transport()

Build the record-replay transport stack for an httpx.AsyncClient.

Usage

Source

llmcache.transport(*, mode=None)

Parameters

mode: LlmCacheMode | None = None: The cache mode; None reads LlmCacheSettings.mode.

Returns

httpx.AsyncBaseTransport: CachingTransport(RetryTransport(httpx.AsyncHTTPTransport())), or a bare
RetryTransport when the mode is LlmCacheMode.BYPASS.

Example

>>> AsyncOpenAI(http_client=httpx.AsyncClient(transport=transport()))

llmcache.LlmCacheMiss

Raised in LlmCacheMode.REPLAY when a request has no recorded response.

Usage

Source

llmcache.LlmCacheMiss()