Compute providers and cloud machines
Prism can power an existing cloud virtual machine on before it starts a model server there, and power it off after idle time. It never creates, rents, resizes or deletes machines. The only calls to the provider read the instance state, start it and stop it. Nebius Compute is built in; other clouds are added by plugin packages (see Other clouds), starting with Amazon EC2 and Google Compute Engine.
1. Create a service account in Nebius
Section titled “1. Create a service account in Nebius”In the Nebius console, create a service account for Prism and grant it only what reading, starting and stopping your instances needs. Do not grant editor or admin roles. Add an authorized key to the service account (Service account → Authorized keys). Keep the private key: it is a long-lived credential.
An Object Storage access key (ID NAKI…) cannot control Compute instances;
Prism names it and refuses it.
2. Prepare the machine
Section titled “2. Prepare the machine”Install the model server and the service agent
as user units, and enable lingering for the agent’s account
(loginctl enable-linger) so the agent answers after every power-on without
a login. For idle power-off, the agent needs
nvidia_smi = "/usr/bin/nvidia-smi" in its configuration (pairing sets it
when that file exists).
Pair the machine as described in Add a machine.
3. Add the cloud account in the console
Section titled “3. Add the cloud account in the console”Compute providers → Cloud accounts → Add cloud account. Enter:
- Service account ID (
serviceaccount-…, under the account’s name in the Nebius console), - Public key ID (
publickey-…, on the service account’s Authorized keys tab), - Private key (paste the PEM, or Or choose the key file).
A credentials JSON from nebius iam auth-public-key generate is accepted too
and split into these fields. Instead of the key you can enter a 1Password
reference (op://Prism/nebius/credentials) when 1Password is configured as
the secrets backend. Prism keeps the key in its encrypted vault; it never
appears in the console, the API, logs or state files.
Press List instances. Prism lists the instances in every region of the account (read only). Pick the instance and press Add as machine, then set:
| Setting | Default | Meaning |
|---|---|---|
| Power-on timeout | 10 minutes | How long a power-on may take until the agent answers |
| Power off after idle | off | Power the machine off after this long without requests to its model servers |
A paired machine with the same host name or address is linked to the cloud machine automatically; otherwise choose it in Linked cloud machine.
What happens
Section titled “What happens”- A request that names a model on the machine waits while Prism powers the machine on, waits for the agent, starts the model server and checks that the model answers. Several waiting requests share one power-on.
- A request where Prism chooses the model goes to another model at once. The machine is powered on in the background only when its model would have been Prism’s first choice.
- Power off after idle: after that long without requests, Prism asks the agent for GPU processes outside the agent’s own units. If there are any, the machine stays on and its card says “Not powered off: other processes are using the GPU” with the processes (name, PID, memory). Otherwise Prism stops the model servers, checks again, and powers the machine off.
- Power off on the machine card powers it off (model servers first) and
stops Prism from powering it on by itself, also across restarts: requests
that name its models get HTTP 503
host_owner_stoppedat once. Power on, or Start on one of its services, allows it again. A machine powered off in the Nebius console or withshutdownblocks nothing.
Errors
Section titled “Errors”| Code | Meaning |
|---|---|
host_power_on_failed |
The cloud did not start the instance |
host_capacity_unavailable |
The cloud has no free capacity (GPU) for the instance now; requests that name its models fail at once instead of waiting for the power-on timeout, and the next request tries again |
host_unreachable |
The instance runs, but the agent did not answer within the power-on timeout |
host_credentials_unavailable |
The key is missing or Nebius rejected it (“The machine access key is not set or was rejected.”) |
host_owner_stopped |
The owner powered the machine off in the console |
After these, the model-server stages apply: start_failed,
readiness_timeout (see
Start on request and stop when idle).
Other clouds from plugin packages
Section titled “Other clouds from plugin packages”A cloud other than Nebius is a signed plugin package with the compute capability. After the owner installs and enables it on the Plugins page (see Installed packages), Add cloud account offers the cloud next to Nebius, with the fields its package declares, their help text and the minimal rights the key needs.
- Prism keeps the field values in its encrypted vault. The package’s worker receives them only while it lists, powers on or powers off that account’s machines, and signs its requests to the cloud itself. Prism sends them only to the addresses the package declared, never follows redirects, and keeps them out of the console, the API, logs and state files.
- A package update cannot change the cloud’s addresses, so a stored key never starts going somewhere new.
- After a power-on Prism reads the machine’s addresses again. When Prism reaches the machine’s agent at the machine’s public address and the cloud gave the machine a new one, Prism moves the agent and its model servers to the new address before it waits for them. An agent reached over a private network, such as a tailnet, is not moved.
- Some clouds bill a stopped machine like a running one. When the package says so for a machine, its settings warn that Power off after idle does not lower the bill.
Amazon EC2
Section titled “Amazon EC2”The Amazon EC2 package lists the instances of the regions you name and starts and stops one instance; it never creates, changes or terminates instances. Create an IAM user for Prism with only this policy and an access key for it:
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ec2:DescribeInstances","ec2:StartInstances","ec2:StopInstances"],"Resource":"*"}]}In the account form enter Access key ID, Secret access key and
Regions (region codes separated by commas, for example
us-east-1, eu-west-1; empty means us-east-1; at most eight). Only
instances with an EBS root volume can be stopped; an instance-store instance
is listed as billed while stopped. When AWS answers
InsufficientInstanceCapacity, the machine shows host_capacity_unavailable.
A stopped instance keeps its EBS volumes (billed) and loses a public address
that is not an Elastic IP.
Google Compute Engine
Section titled “Google Compute Engine”The Google Compute Engine package lists the instances of one project and starts and stops one instance; it never creates, changes or deletes instances. Create a custom role for Prism with only these permissions:
compute.instances.list, compute.instances.get, compute.instances.start,compute.instances.stop, compute.zoneOperations.getCreate a service account, grant it that role on the project, and create a
JSON key for it (Keys → Add key → Create new key → JSON). In
the account form paste the whole key file into Service account key
(JSON); Project ID (empty means the project in the key file) and
Zones (for example us-central1-a, europe-west4-b; empty means every
zone; at most 16) are optional. An API key or an OAuth client file is
refused with a message that names the mistake.
For each operation the package signs a short-lived token request with the
key, and only the token, never the key, goes to the Compute Engine API. A
start waits for Google’s operation: when the zone has no free GPUs or the
project’s GPU quota is used up (ZONE_RESOURCE_POOL_EXHAUSTED,
QUOTA_EXCEEDED), the machine shows host_capacity_unavailable at once. A
stopped instance keeps its disks and static addresses (billed) and loses an
ephemeral external address. GPU instances cannot be suspended, only
stopped.
Microsoft Azure
Section titled “Microsoft Azure”The Microsoft Azure package lists the virtual machines of the subscriptions
you name and starts and deallocates one machine; it never creates, changes or
deletes machines. Register an application for Prism (Microsoft Entra ID →
App registrations), create a client secret for it, and give it only this
custom role on the subscription (or on the resource group of your GPU
machines), with your subscription ID in place of SUBSCRIPTION_ID:
{"properties":{"roleName":"Prism power","permissions":[{"actions":["Microsoft.Compute/virtualMachines/read","Microsoft.Compute/virtualMachines/instanceView/read","Microsoft.Compute/virtualMachines/start/action","Microsoft.Compute/virtualMachines/deallocate/action","Microsoft.Network/networkInterfaces/read","Microsoft.Network/publicIPAddresses/read"]}],"assignableScopes":["/subscriptions/SUBSCRIPTION_ID"]}}In the account form enter Directory (tenant) ID, Application (client)
ID, Client secret (its Value, not its ID) and Subscription IDs
(separated by commas; at most eight). The client secret goes only to
login.microsoftonline.com for a token; the requests to Azure Resource Manager
carry that token.
Power-off always deallocates the machine: a machine that is only powered
off stays allocated and Azure bills it as running. A deallocated machine keeps
its disks (billed) and loses a dynamic public IP address; Prism reads the new
one after the next start. Azure accepts a start before it finds hardware; when
the allocation then fails (AllocationFailed, ZonalAllocationFailed,
OverconstrainedAllocationRequest, SkuNotAvailable or a quota error), the
machine shows host_capacity_unavailable and waiting requests fail at once.
A machine in a resource group whose name contains parentheses is not listed.
Crusoe Cloud
Section titled “Crusoe Cloud”The Crusoe Cloud package lists the virtual machines of one project and starts
and stops one machine; it never creates, changes or deletes machines. Crusoe
Cloud API keys act with the rights of the user who creates them, and the
narrowest role that can start and stop machines is project-editor. Give
Prism its own user (Admin → User Access → Team → Invite User) with no
organization role and only project-editor on the project of your GPU
machines, and as that user create a key under Security → Cloud API keys.
In the account form enter Access key, Secret key (shown once) and Project ID. The package signs each request with the secret key; the key itself is never sent.
A stopped on-demand machine is not billed for compute. A machine under a
reservation or an unexpired commitment is billed whether it runs or not; its
settings warn that Power off after idle does not lower the bill. Public
addresses are dynamic by default and change at every stop and start; Prism
reads the new one after the next start. Crusoe accepts a start and runs it in
the background; when that start fails for lack of capacity, the machine shows
host_capacity_unavailable and waiting requests fail at once.
Scaleway Instances
Section titled “Scaleway Instances”The Scaleway Instances package lists the servers of the zones you name and powers one on and off; it never creates, changes or deletes servers. Create an IAM application for Prism (IAM → Applications), a policy attached to it with one rule on the project of your GPU machines and only these permission sets, then generate an API key for the application:
InstancesServerList, InstancesServerRead, InstancesServerStart, InstancesServerStopIn the account form enter Secret key (the API key’s secret key, not the
access key that starts with SCW), optionally Project ID (list only that
project’s servers) and Zones (zone codes separated by commas, for example
fr-par-2; empty means every Instance zone). Requests go only to
api.scaleway.com.
Power-off always uses power off, never stop in place or standby: those keep
the hardware allocated and Scaleway bills it as running. A powered-off server
keeps its volumes and flexible IPs (billed) and loses a dynamic public IP
address; Prism reads the new one after the next start. The scratch disk of
an H100 or L40S server is erased at power-off. When Scaleway refuses a start
(out_of_stock, quotas_exceeded), the machine shows
host_capacity_unavailable and waiting requests fail at once.
Alibaba Cloud ECS
Section titled “Alibaba Cloud ECS”The Alibaba Cloud ECS package lists the instances of the regions you name and starts and stops one instance; it never creates, changes or releases instances. Create a RAM user for Prism with permanent AccessKey access and give it only this policy:
{"Version":"1","Statement":[{"Effect":"Allow","Action":["ecs:DescribeInstances","ecs:StartInstance","ecs:StopInstance"],"Resource":"*"}]}In the account form enter AccessKey ID, AccessKey secret and
Regions (region IDs such as cn-hangzhou, ap-southeast-1, separated by
commas; at most eight). The package signs each request with the AccessKey
secret (signature V3); the secret itself is never sent.
Power-off stops the instance in economical mode: a pay-as-you-go instance
in a VPC without local disks stops billing for vCPUs, memory and GPUs, while
its cloud disks and EIP are still billed and a public IP that is not an EIP is
released; Prism reads the new one after the next start. Alibaba Cloud keeps
billing a subscription instance, an instance in the classic network and an
instance with local disks (the d and i families) when it is stopped; their
settings warn that Power off after idle does not lower the bill. Because
an instance in economical mode gives its hardware back, the zone may have none
left when it starts again (OperationDenied.NoStock); the machine then shows
host_capacity_unavailable and waiting requests fail at once.
Paperspace
Section titled “Paperspace”The Paperspace package lists the machines of one Paperspace team and starts
and stops one machine; it never creates, changes or deletes machines. In the
Paperspace console open the profile icon → Team settings → API keys,
enter a name (for example prism), click Add and copy the key, which is
shown only once. Paste it into API key in the account form. A Paperspace
key acts for the whole team and cannot be limited to listing, starting and
stopping machines, so give Prism a key of its own that you can delete
separately. The key goes only to api.paperspace.com, in the
Authorization header.
A stopped machine is not billed for compute; its storage and a static public
IP address stay billed. A dynamic public IP address is released when the
machine is off; Prism reads the new one after the next start. When Paperspace
accepts a start and then cannot finish it for lack of hardware, the machine
shows host_capacity_unavailable and waiting requests fail at once.
Oblivus
Section titled “Oblivus”The Oblivus package lists the virtual machines of an Oblivus Cloud account and starts and stops one machine; it never deploys, changes or deletes machines. In the Oblivus console open API → Credentials and generate an API key for Prism with only these permissions:
virtual-server:list, virtual-server:details, virtual-server:start,virtual-server:stopIn the account form enter the API key; Prism does not use the API token
that Oblivus shows next to it. The key goes only to api.oblivus.com.
A stopped machine is billed for its storage and IP address only, and keeps its
IP address. The listing shows each machine’s running price per hour; a machine
whose stopped price equals its running price is marked as billed while
stopped. Oblivus sends the whole machine list in one answer, so an account
with more than about 30 machines does not fit Prism’s 64 KiB answer limit and
its listing fails with a message that says so. Oblivus does not document the
answer to a start without free hardware; a refusal that mentions stock or
capacity shows host_capacity_unavailable.
Hyperstack
Section titled “Hyperstack”The Hyperstack package lists the virtual machines of a Hyperstack account,
hibernates one machine and restores it; it never creates, resizes or deletes
machines. Create the API key in the Hyperstack console under API Keys. A
key acts with the role of the user who created it, so create it as an
organization member whose role allows only viewing, hibernating and restoring
virtual machines, plus billing:view and billing:credit-view, which
Hyperstack checks before a restore. In the account form enter the API key;
it goes only to infrahub-api.nexgencloud.com.
Power-off is always hibernation: a machine stopped in place (SHUTOFF)
keeps its hardware and Hyperstack bills it as running, while a hibernated
machine is billed for its root disk only. Prism never uses the plain stop. A
flavor that cannot hibernate (labelled no-hibernation: spot flavors,
large-root-disk flavors and some GPU families such as B200-SXM) is marked as
billed while stopped, and Prism leaves such a machine on. A hibernated machine
loses its public IP address; Prism reads the new one after the restore. Its
hardware is not reserved either: a restore needs the flavor in stock, and a
refusal that mentions stock or capacity shows host_capacity_unavailable.
Oracle Cloud (OCI Compute)
Section titled “Oracle Cloud (OCI Compute)”The OCI package lists the Compute instances of the regions and compartments
you name and starts and stops one instance; it never launches, changes or
terminates instances. It needs Prism 0.37.1 or later. Create a user for Prism
in a group, for example prism-power, and give the group only this policy
(replace COMPARTMENT with the compartment’s name, or write in tenancy):
Allow group prism-power to read instances in compartment COMPARTMENTAllow group prism-power to use instances in compartment COMPARTMENT where request.permission = 'INSTANCE_POWER_ACTIONS'Allow group prism-power to read vnic-attachments in compartment COMPARTMENTAllow group prism-power to read vnics in compartment COMPARTMENTOn the user’s page add an API key (API keys → Add API key →
Download private key → Add) and copy the Configuration file
preview into the account form: Tenancy OCID, User OCID,
Fingerprint, Private key (the whole downloaded PEM, without a
passphrase), Regions (at most 8) and optionally Compartment OCIDs
(empty means the root compartment; regions times compartments at most 8).
Prism checks that the fingerprint belongs to the key; the key itself never
leaves Prism, only signatures go to iaas.<region>.oraclecloud.com. Twelve of
Oracle’s 44 commercial regions are not reachable because a package declares
at most 32 addresses: ap-batam-1, ap-kulai-2, af-casablanca-1,
sa-bogota-1, sa-valparaiso-1, eu-turin-1, eu-madrid-3,
ap-singapore-2, ap-chuncheon-1, uk-cardiff-1, mx-monterrey-1 and
me-jeddah-1.
Power-off is a soft stop: the operating system shuts down, and OCI powers the
instance off after 15 minutes at most. A stopped instance whose shape holds
GPUs (all except the VM.GPU.A10 series), local NVMe (Dense I/O, HPC) or a
whole Optimized bare metal host is still billed in full; the console marks it
and warns that powering it off does not lower the bill. Other shapes stop
billing for compute and keep billing their volumes; a stopped instance keeps
its ephemeral public IP. When OCI answers a start with “Out of host capacity”
or a service limit, the machine shows host_capacity_unavailable and waiting
requests fail at once. The listing reads addresses for running instances
within Prism’s 32 calls per listing; the rest show their addresses once
their state is read.
Configure in config.toml instead
Section titled “Configure in config.toml instead”[secrets]backend = "op-cli" # only for credentials_ref
[[inference.hosts]]id = "gpu-1"provider = "nebius"instance_id = "computeinstance-e00example"credentials_ref = "op://Prism/nebius/credentials" # or:# credentials_env = "PRISM_NEBIUS_CREDENTIALS" # the credentials JSON itselfpower_on_timeout = "10m"# power_off_after = "30m"
[[inference.services]]# ... as in the service agent page, plus:host_id = "gpu-1"Exactly one of credentials_ref and credentials_env is required. Plugin
clouds are added in the console only. See the
inference configuration reference.