Skip to content

[runtime_policy]

Timeouts of upstream requests and streams, and internal buffer sizes.

Key Type Default Description
runtime_policy.health_max_consecutive_failures integer health_max_consecutive_failures is how many failed health checks in a row mark a provider unavailable (default 3).
runtime_policy.rerank_default_top_n integer rerank_default_top_n is the top_n Prism sends when a rerank request has none (default 64).
runtime_policy.rerank_max_top_n integer rerank_max_top_n is the largest top_n a rerank request may ask for (default 128); larger values get 400 invalid_request_size.
runtime_policy.stream_open_timeout_ms integer stream_open_timeout_ms is the self-hosted/local upstream open-phase timeout override. Kept for compatibility with earlier drafts before remote/local split.
runtime_policy.stream_open_timeout_remote_ms integer Stream open phase (upstream HTTP 200 not yet received). Defaults: 8000 ms for remote providers, 60000 ms for remote models that reason: their provider or another seller of the same model says so, or an answer carried reasoning (the provider answers with headers only after the backend has read the prompt), 90000 ms for self-hosted/local ones. stream_open_timeout_ms is the self-hosted/local value, not a global one.
runtime_policy.stream_open_timeout_remote_reasoning_ms integer stream_open_timeout_remote_reasoning_ms is the open-phase timeout of a stream to a remote model that reasons before answering (default 60000 ms): an aggregator sends its headers only after the backend has read the prompt.
runtime_policy.stream_idle_timeout_remote_ms integer Silence on a committed stream body. 0 disables. Defaults match the non-stream attempt deadlines (120s remote, 600s local).
runtime_policy.stream_idle_timeout_local_ms integer stream_idle_timeout_remote_ms and stream_idle_timeout_local_ms end a stream whose body stays silent this long, for remote and self-hosted providers (defaults 120000 and 600000 ms; 0 disables; negative fails load).
runtime_policy.upstream_request_timeout_remote_ms integer Total deadline for each non-stream provider attempt. Explicit values must be positive. Defaults: remote/cloud 120000 ms; remote models that reason (as above) 600000 ms; self-hosted/local 600000 ms.
runtime_policy.upstream_request_timeout_remote_reasoning_ms integer upstream_request_timeout_remote_reasoning_ms is the deadline of one non-stream attempt to a remote model that reasons before answering (default 600000 ms).
runtime_policy.upstream_request_timeout_local_ms integer upstream_request_timeout_local_ms is the deadline of one non-stream attempt to a self-hosted provider (default 600000 ms).