Serve
serve.RapidMlxSettings
The [serve.rapid-mlx] section: the daily-pinned rapid-mlx version, model, and port.
Usage
serve.RapidMlxSettings()The default text lane. On ordinary tool calling it and LlamaServerSettings are equivalent: serving the same Qwen3.6-35B-A3B to both, 42 calls each, every call dispatched and every argument came back valid and schema-conformant — flat, enum, nested, arrays of objects, multi-tool selection, and numeric-verbatim alike. Pick between them on the two edges below, not on general quality.
rapid-mlx serves MLX-native weights only, and honors tool_choice="required".
Warning
rapid-mlx extracts tool calls with a delimiter-based parser (it advertises qwen3_coder_xml), and a string argument containing that delimiter is truncated at it: send </function> inside a payload and the argument arrives silently short, as well-formed JSON no downstream check can catch. Serve LlamaServerSettings when arguments can carry tool-call syntax — quoted model output, agent transcripts, scraped markup.
serve.MlxVlmSettings
The [serve.mlx-vlm] section: the daily-pinned mlx-vlm version, vision model, and port.
Usage
serve.MlxVlmSettings()serve.LlamaServerSettings
The [serve.llama-server] section: a full llama-server command string and its port.
Usage
serve.LlamaServerSettings()The opt-in GGUF lane, equivalent to the default RapidMlxSettings on ordinary tool calling. Its one advantage is string fidelity: it returns arguments containing tool-call delimiter syntax verbatim, where rapid-mlx’s parser truncates them. That is the reason to select it.
Takes a full command string rather than a model and port, because a llama-server invocation carries flags no recipe can generalize.
Warning
llama-server ignores tool_choice="required" — it applies no grammar constraint, so instead of forcing a call it can answer in prose and run to max_tokens, returning no tool call at all. Code that depends on being handed one must stay on rapid-mlx or treat a missing call as a real outcome.
serve.ManagedServer
A recipe-configured OpenAI-compatible server with lifecycle + health, over any backend.
Usage
serve.ManagedServer(recipe, *, model=None, port=None)The recipe picks the backend: a local uvx subprocess (rapid-mlx, mlx-vlm, llama-server) or a scale-to-zero hosted Modal vLLM endpoint (modal-vllm). Health, model verification, readiness, and the client are identical across both — only where the endpoint lives and how it starts and stops differ, which is all a ServeBackend answers.
model and port override the recipe’s configured pair for one server — serving an arbitrary artifact (a freshly fused MLX directory, say) without touching the process-global settings. An overridden server is a distinct managed process: it carries its own detach run name and launchd label, so it neither adopts nor tears down the config-driven one.
Parameter Attributes
recipe: Recipemodel: str | None = Noneport: int | None = None
Example
>>> server = ManagedServer("rapid-mlx")
>>> handle = await server.ensure()
>>> fused = ManagedServer("rapid-mlx", model="/runs/watcher/fused", port=8410)Attributes
| Name | Description |
|---|---|
| api_key | The bearer credential that reaches this server’s endpoint. |
| backend | The substrate this recipe runs on: a local subprocess, or a hosted Modal endpoint. |
| base_url |
The OpenAI-compatible /v1 endpoint this server answers on.
|
| identity | What distinguishes an overridden server: the port it listens on and the model it serves. |
| label | The launchd label owning this server’s agent. |
| required_models | Every model id the endpoint must report, not just the base. |
| run_name | The detach run name owning this server’s process. |
| served_model | The model this server serves: the override, else the recipe’s configured model. |
| served_port |
The port this server listens on: the override, else the configured port; None when hosted.
|
api_key
The bearer credential that reaches this server’s endpoint.
api_key: str
backend
The substrate this recipe runs on: a local subprocess, or a hosted Modal endpoint.
backend: ServeBackend
base_url
The OpenAI-compatible /v1 endpoint this server answers on.
base_url: str
identity
What distinguishes an overridden server: the port it listens on and the model it serves.
identity: str
The model belongs in the identity, not just the port: two runs that land on one port would otherwise share a run name and a launchd label, and each would tear down the other’s process while verifying it served the wrong model.
label
The launchd label owning this server’s agent.
label: str
required_models
Every model id the endpoint must report, not just the base.
required_models: set[str]
An override serves exactly the one model it names. A hosted vLLM recipe serves its base model and the LoRA adapter it exists to score under a second id — an endpoint serving only the base would pass a base-only identity check while missing the adapter. A llama-server command advertises nothing checkable, so it requires none.
run_name
The detach run name owning this server’s process.
run_name: str
served_model
The model this server serves: the override, else the recipe’s configured model.
served_model: str | None
served_port
The port this server listens on: the override, else the configured port; None when hosted.
served_port: int | None
A hosted recipe answers on a workspace-scoped HTTPS URL, not a loopback port, so it has none.
Methods
| Name | Description |
|---|---|
| client() |
Return an AsyncOpenAI client bound to this recipe’s endpoint.
|
| ensure() | Return a healthy server for this recipe, starting it if necessary. |
| health() |
Return True when GET /v1/models answers 2xx within the backend’s probe timeout.
|
| idle() |
An ~athome.idle.IdleResource that spawns this server on first use and stops it once idle.
|
| probe() | Report health without waking a scale-to-zero backend — a passive, side-effect-free check. |
| stop() | Stop this recipe by tearing down its backend. |
| verify_served_model() |
Raise ServeError unless GET /v1/models lists every model this server requires.
|
client()
Return an AsyncOpenAI client bound to this recipe’s endpoint.
Usage
client(*, cached=False)Parameters
cached: bool = False- Route requests through athome.llmcache.transport() for record/replay.
Returns
AsyncOpenAI-
A client with base_url set to the recipe’s
/v1endpoint and its api key.
ensure()
Return a healthy server for this recipe, starting it if necessary.
Usage
ensure(*, persistent=False)A recipe already answering healthy returns its handle without restarting. Otherwise the backend brings it up — a local recipe launches its command vector, detached via athome.detach (persistent=False) or as a launchd KeepAlive agent (persistent=True); a hosted recipe deploys its Modal app — and the call blocks until health polling succeeds.
Parameters
persistent: bool = False- Install a launchd KeepAlive agent instead of a detached run (local recipes only).
Returns
ServerHandle- The running server’s handle.
Raises
ServeError-
The adopted or launched server serves a different model.
HealthTimeout- The server did not report healthy within the ready timeout.
health()
Return True when GET /v1/models answers 2xx within the backend’s probe timeout.
Usage
health()idle()
An ~athome.idle.IdleResource that spawns this server on first use and stops it once idle.
Usage
idle(*, ttl_s)Each ~athome.idle.IdleResource.use() ensures the server is healthy and yields its handle; the resource’s reaper stops the server once it has sat unused past ttl_s.
Parameters
ttl_s: float- Idle seconds before the reaper stops the server.
Returns
IdleResource[ServerHandle]- A resource wired to ensure() and stop().
probe()
Report health without waking a scale-to-zero backend — a passive, side-effect-free check.
Usage
probe()A local recipe probes its loopback endpoint (free); a hosted recipe observes its deployment through Modal’s control plane, never an HTTP request that would cold-boot and bill a container.
stop()
Stop this recipe by tearing down its backend.
Usage
stop()A local recipe boots out its launchd agent and kills its detached process group — the two run independently, so a launchd uninstall failure never skips the process kill. A hosted recipe has no local process: Modal’s scaledown reaps it, so this is a no-op.
verify_served_model()
Raise ServeError unless GET /v1/models lists every model this server requires.
Usage
verify_served_model()This is an identity check, not authentication: a process holding the endpoint could spoof the model ids, and trusting the resolved endpoint is inherent to the design.
serve.ServerHandle
A running recipe’s port (None for a hosted endpoint), pid (None for a launchd
Usage
serve.ServerHandle(recipe, port, pid, base_url, api_key)service or a hosted endpoint), OpenAI base URL, and the bearer credential that reaches it.
api_key is "local" for a loopback recipe and the recipe’s configured key for a hosted one, so a consumer holding only the handle can authenticate against the endpoint.
Parameter Attributes
recipe: Recipeport: int | Nonepid: int | Nonebase_url: strapi_key: str
Example
>>> handle = await up("mlx-vlm")
>>> handle.base_url‘http://127.0.0.1:8401/v1’
serve.up()
Ensure recipe is running and healthy, returning its handle.
Usage
serve.up(recipe, *, persistent=False)serve.down()
Stop recipe — launchd uninstall or detached-process kill.
Usage
serve.down(recipe)serve.settings_for()
Usage
serve.settings_for(recipe)serve.command_for()
The recipe’s command vector; model and port override its configured pair.
Usage
serve.command_for(recipe, *, model=None, port=None)serve.probe_all()
Report the health of every configured recipe, without waking a scale-to-zero endpoint.
Usage
serve.probe_all()serve.ServeError
Raised when a managed server cannot be started, stopped, or reached.
Usage
serve.ServeError()serve.HealthTimeout
Raised when a spawned server does not report healthy within the ready timeout.
Usage
serve.HealthTimeout()