[inference]
Model servers and machines that Prism starts, stops and powers (lifecycle control needs Pro).
| Key | Type | Default | Description |
|---|---|---|---|
[[inference.services]] |
array of tables | services are model servers Prism may start and stop: over SSH as a systemd unit, or through prism-service-agent on the model machine. Starting and stopping need Prism Pro. |
|
inference.services[].id |
string | ID names the service in the console and in commands. | |
inference.services[].label |
string | name shown in the console | |
inference.services[].provider |
string | provider is the [[providers]] entry the service serves; it must exist, use the same host name and be self_hosted. |
|
inference.services[].model |
string | model is the model id the service serves. |
|
inference.services[].transport |
string | Agent transport instead of SSH (prism-service-agent on the model host, see . ./deploy/SERVICE-AGENT.md). With transport = “agent” leave host, port, user, unit, system, identity_file and known_hosts_file unset. ssh (default) | agent | |
inference.services[].host |
string | host is the SSH host of the service (ssh transport). |
|
inference.services[].port |
integer | port is the SSH port (default 22). |
|
inference.services[].user |
string | user is the SSH user. |
|
inference.services[].unit |
string | unit is the systemd unit that runs the model server. |
|
inference.services[].system |
bool | system is true for a system unit, false for a user unit. |
|
inference.services[].identity_file |
string | identity_file and known_hosts_file are absolute paths of the SSH key and known-hosts file. |
|
inference.services[].known_hosts_file |
string | identity_file and known_hosts_file are absolute paths of the SSH key and known-hosts file. |
|
inference.services[].agent_url |
string | agent_url is the address of prism-service-agent on the model machine (agent transport); its host must match the provider’s host name. |
|
inference.services[].agent_token_env |
string | agent_token_env names the environment variable holding the agent’s token. |
|
inference.services[].agent_service |
string | service id in the agent’s allowlist | |
inference.services[].dedicated |
bool | dedicated must be true: the service serves this one model. |
|
inference.services[].autostart |
bool | autostart lets a client request start a stopped service (Pro and above). |
|
inference.services[].autostart_max_waiters |
integer | autostart_max_waiters bounds requests waiting for one start; 0 means 32. |
|
inference.services[].start_timeout |
integer | start_timeout bounds one start, manual or automatic; 0 means the manager default (3 minutes). |
|
inference.services[].autostart_wait |
integer | autostart_wait bounds how long one explicit request waits; 0 means start_timeout. |
|
inference.services[].idle_stop_after |
integer | idle_stop_after stops a ready service after this long without requests; 0 disables idle stop. |
|
inference.services[].host_id |
string | host_id names the [[inference.hosts]] entry Prism powers on before starting this service. Empty means the host is always on. |
|
[[inference.hosts]] |
array of tables | hosts are existing cloud machines Prism may power on and off for the services that name them with host_id. |
|
inference.hosts[].id |
string | ID names the host; a service refers to it with host_id. | |
inference.hosts[].provider |
string | provider is the cloud provider of the machine; nebius is the only one supported. |
|
inference.hosts[].instance_id |
string | instance_id is the cloud provider’s id of the existing machine. Prism powers it on and off; it never creates, resizes or deletes machines. |
|
inference.hosts[].credentials_ref |
string | needs [secrets] backend | |
inference.hosts[].credentials_env |
string | or the JSON in env; one of the two | |
inference.hosts[].power_on_timeout |
integer | power_on_timeout bounds one power-on until the service host answers; 0 means 10 minutes. |
|
inference.hosts[].power_off_after |
integer | power_off_after powers the host off after all its services were idle or stopped for this long and its agent reports no foreign GPU process; 0 (the default) never powers it off automatically. |
|
inference.console |
table of interface | console is what the console’s compute setup keeps: paired machines with their certificate pin, managed services, cloud accounts and powered hosts. The setup reads and writes it; the loader only carries it. |