Managed inference
Managed inference lets Prism control model servers that you already run:
a llama-server, vLLM, Ollama or SGLang process on your own machine or on an
existing cloud GPU machine. Prism does not download models, rent machines or
create virtual machines.
What Prism does
Section titled “What Prism does”- Starts and stops a model server. Each model server is a systemd unit
that serves one model through an OpenAI-compatible API. Prism starts the
unit, waits until the model answers on
/healthand/v1/models, and only then sends requests to it. On stop, Prism stops sending new requests, lets running ones finish, then stops the unit. - Starts a model server when a request needs it and stops it after a period without requests. See Start on request and stop when idle.
- Powers a cloud machine on and off. For a Nebius Compute instance, Prism powers the machine on before it starts a model server there and can power it off after idle time. See Compute providers and cloud machines.
How Prism reaches a machine
Section titled “How Prism reaches a machine”| Transport | How it works | Set up |
|---|---|---|
| Service agent (recommended) | prism-service-agent runs on the model machine as the account that owns the model servers. Prism calls it over HTTPS with a token. No SSH, no root. |
Compute providers → Add machine in the console, one command on the machine. See Add a machine. |
| SSH | Prism logs in with a dedicated key and runs systemctl --user (or the system manager) for one unit. |
[[inference.services]] in config.toml. See the inference configuration reference. |
Where to manage it
Section titled “Where to manage it”Open Compute providers in the console sidebar (owner only). The page lists machines, the model servers on each machine, and cloud accounts. Every model server shows its state (Stopped, Starting, Ready, Completing requests, Stopping), the model, the number of active requests and the Start and Stop buttons.
Entries from config.toml appear on the page marked from config and can
only be changed in config.toml.
When the subscription is not active
Section titled “When the subscription is not active”- New Start and Stop commands, monitoring and automatic start or power-off are refused. The page says why (for example “No subscription is connected.”, or, below Pro, “This feature needs Prism Pro. Please upgrade your plan.”).
- Model servers that are running keep running and keep serving requests. Prism never stops a server or powers off a machine because the subscription ended.
- A request that names a stopped model gets HTTP 503 with the reason
license_required.
Agents can manage it too
Section titled “Agents can manage it too”An agent can add model servers and start, stop and power machines through Prism with a scoped token instead of SSH. See Agent access tokens.