Skip to content

Managed inference

Managed inference lets Prism control model servers that you already run: a llama-server, vLLM, Ollama or SGLang process on your own machine or on an existing cloud GPU machine. Prism does not download models, rent machines or create virtual machines.

  • Starts and stops a model server. Each model server is a systemd unit that serves one model through an OpenAI-compatible API. Prism starts the unit, waits until the model answers on /health and /v1/models, and only then sends requests to it. On stop, Prism stops sending new requests, lets running ones finish, then stops the unit.
  • Starts a model server when a request needs it and stops it after a period without requests. See Start on request and stop when idle.
  • Powers a cloud machine on and off. For a Nebius Compute instance, Prism powers the machine on before it starts a model server there and can power it off after idle time. See Compute providers and cloud machines.
Transport How it works Set up
Service agent (recommended) prism-service-agent runs on the model machine as the account that owns the model servers. Prism calls it over HTTPS with a token. No SSH, no root. Compute providers → Add machine in the console, one command on the machine. See Add a machine.
SSH Prism logs in with a dedicated key and runs systemctl --user (or the system manager) for one unit. [[inference.services]] in config.toml. See the inference configuration reference.

Open Compute providers in the console sidebar (owner only). The page lists machines, the model servers on each machine, and cloud accounts. Every model server shows its state (Stopped, Starting, Ready, Completing requests, Stopping), the model, the number of active requests and the Start and Stop buttons.

Entries from config.toml appear on the page marked from config and can only be changed in config.toml.

  • New Start and Stop commands, monitoring and automatic start or power-off are refused. The page says why (for example “No subscription is connected.”, or, below Pro, “This feature needs Prism Pro. Please upgrade your plan.”).
  • Model servers that are running keep running and keep serving requests. Prism never stops a server or powers off a machine because the subscription ended.
  • A request that names a stopped model gets HTTP 503 with the reason license_required.

An agent can add model servers and start, stop and power machines through Prism with a scoped token instead of SSH. See Agent access tokens.