Skip to content

Start on request and stop when idle

Both settings are off by default and are set per model service.

In the console: Compute providers → the service → Edit → Start on request, and optionally Stop after idle. Or in config.toml:

[[inference.services]]
id = "qwen-coder"
provider = "gpu-coder"
model = "qwen-coder" # exact ID returned by /v1/models
transport = "agent"
agent_url = "https://192.0.2.10:18093"
agent_token_env = "PRISM_GPU_CODER_AGENT_TOKEN"
agent_service = "qwen-coder"
dedicated = true
autostart = true
start_timeout = "10m" # longer than the model's real load time
# autostart_wait = "10m" # default: start_timeout
# autostart_max_waiters = 32 # default 32
# idle_stop_after = "30m" # default: off
Setting Console label Default Meaning
autostart Start on request off A request that names this model starts the stopped server
start_timeout Start timeout 3 minutes How long a start may take, until the model answers; applies to the Start button too
autostart_wait Request wait start_timeout How long a request waits for the start
autostart_max_waiters Waiting requests limit 32 Requests that may wait at the same time
idle_stop_after Stop after idle off Stop the server after this long without requests
  • A request that names the model (for example gpu-coder::qwen-coder, or qwen-coder) waits while the model loads and is then answered by that model, never by another one. When it waits longer than autostart_wait, or more than autostart_max_waiters requests are waiting, it gets HTTP 503 with Retry-After.
  • A request where Prism chooses the model (auto, a role alias, a routing profile) does not wait. It goes to another eligible model, and the server starts in the background only when Prism would have chosen it first.
  • Console views, /v1/models, catalog refresh and health probes never start a server.

While the model loads, every request that names it waits. Plan loads for times when that is acceptable, or leave autostart off.

  • Stop in the console blocks autostart until you press Start, also across Prism restarts. The service shows “Stopped by the owner: autostart waits for the Start button”.
  • An idle stop, or a stop outside Prism (systemctl --user stop, a crash, a reboot), does not block autostart.

The service row shows whether autostart is on, how many requests wait, and why the service stopped.

Stopping a model server does not power off its machine or end cloud billing. A cloud machine is powered off only by its own idle setting; see Compute providers and cloud machines.

State or code Meaning What to do
start_failed The unit did not start Read the unit’s log on the machine (journalctl --user -u llama-coder.service)
readiness_timeout The model did not answer within start_timeout Raise start_timeout above the real load time
catalog_refresh_failed The server answered, but Prism could not read its model list after the start Check that /v1/models on the server returns exactly the model set in model
readiness_lost A ready server stopped answering Check the server process
drain_timeout Requests were still running 5 minutes after Stop; the server keeps running with no new requests Finish or cancel the clients, then press Stop again
HTTP 503 license_required The subscription is not active, so the stopped model was not started Renew the subscription, or start the server yourself