Skip to content

[inference]

Model servers and machines that Prism starts, stops and powers (lifecycle control needs Pro).

Key Type Default Description
[[inference.services]] array of tables services are model servers Prism may start and stop: over SSH as a systemd unit, or through prism-service-agent on the model machine. Starting and stopping need Prism Pro.
inference.services[].id string ID names the service in the console and in commands.
inference.services[].label string name shown in the console
inference.services[].provider string provider is the [[providers]] entry the service serves; it must exist, use the same host name and be self_hosted.
inference.services[].model string model is the model id the service serves.
inference.services[].transport string Agent transport instead of SSH (prism-service-agent on the model host, see . ./deploy/SERVICE-AGENT.md). With transport = “agent” leave host, port, user, unit, system, identity_file and known_hosts_file unset. ssh (default) | agent
inference.services[].host string host is the SSH host of the service (ssh transport).
inference.services[].port integer port is the SSH port (default 22).
inference.services[].user string user is the SSH user.
inference.services[].unit string unit is the systemd unit that runs the model server.
inference.services[].system bool system is true for a system unit, false for a user unit.
inference.services[].identity_file string identity_file and known_hosts_file are absolute paths of the SSH key and known-hosts file.
inference.services[].known_hosts_file string identity_file and known_hosts_file are absolute paths of the SSH key and known-hosts file.
inference.services[].agent_url string agent_url is the address of prism-service-agent on the model machine (agent transport); its host must match the provider’s host name.
inference.services[].agent_token_env string agent_token_env names the environment variable holding the agent’s token.
inference.services[].agent_service string service id in the agent’s allowlist
inference.services[].dedicated bool dedicated must be true: the service serves this one model.
inference.services[].autostart bool autostart lets a client request start a stopped service (Pro and above).
inference.services[].autostart_max_waiters integer autostart_max_waiters bounds requests waiting for one start; 0 means 32.
inference.services[].start_timeout integer start_timeout bounds one start, manual or automatic; 0 means the manager default (3 minutes).
inference.services[].autostart_wait integer autostart_wait bounds how long one explicit request waits; 0 means start_timeout.
inference.services[].idle_stop_after integer idle_stop_after stops a ready service after this long without requests; 0 disables idle stop.
inference.services[].host_id string host_id names the [[inference.hosts]] entry Prism powers on before starting this service. Empty means the host is always on.
[[inference.hosts]] array of tables hosts are existing cloud machines Prism may power on and off for the services that name them with host_id.
inference.hosts[].id string ID names the host; a service refers to it with host_id.
inference.hosts[].provider string provider is the cloud provider of the machine; nebius is the only one supported.
inference.hosts[].instance_id string instance_id is the cloud provider’s id of the existing machine. Prism powers it on and off; it never creates, resizes or deletes machines.
inference.hosts[].credentials_ref string needs [secrets] backend
inference.hosts[].credentials_env string or the JSON in env; one of the two
inference.hosts[].power_on_timeout integer power_on_timeout bounds one power-on until the service host answers; 0 means 10 minutes.
inference.hosts[].power_off_after integer power_off_after powers the host off after all its services were idle or stopped for this long and its agent reports no foreign GPU process; 0 (the default) never powers it off automatically.
inference.console table of interface console is what the console’s compute setup keeps: paired machines with their certificate pin, managed services, cloud accounts and powered hosts. The setup reads and writes it; the loader only carries it.